2025-11-03T16:14:04.3196647Z Current runner version: '2.329.0' 2025-11-03T16:14:04.3201155Z Runner name: 'i-0d3c8af4c7ead8235' 2025-11-03T16:14:04.3201703Z Runner group name: 'default' 2025-11-03T16:14:04.3202626Z Machine name: 'ip-10-0-34-62' 2025-11-03T16:14:04.3204718Z ##[group]GITHUB_TOKEN Permissions 2025-11-03T16:14:04.3206854Z Contents: read 2025-11-03T16:14:04.3207287Z Metadata: read 2025-11-03T16:14:04.3207688Z ##[endgroup] 2025-11-03T16:14:04.3209396Z Secret source: Actions 2025-11-03T16:14:04.3210130Z Prepare workflow directory 2025-11-03T16:14:04.3583829Z Prepare all required actions 2025-11-03T16:14:04.3614757Z Getting action download info 2025-11-03T16:14:04.6518123Z Download action repository 'pytorch/test-infra@main' (SHA:dd72a20e27a9820d8aba7291a7d9c1e023f652e1) 2025-11-03T16:14:06.4383924Z Download action repository 'pytorch/pytorch@main' (SHA:94f2657c4b534136aa8958bc35d44ceac5ccd60c) 2025-11-03T16:14:23.5688996Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-11-03T16:14:23.9184139Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-11-03T16:14:24.1575995Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-11-03T16:14:24.3828885Z Download action repository 'seemethere/download-artifact-s3@1da556a7aa0a088e3153970611f6c432d58e80e6' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-11-03T16:14:24.6340882Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-11-03T16:14:24.8967046Z Getting action download info 2025-11-03T16:14:25.0492361Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-11-03T16:14:25.2887079Z Getting action download info 2025-11-03T16:14:25.4050144Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-11-03T16:14:25.6392192Z Getting action download info 2025-11-03T16:14:25.7724265Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-11-03T16:14:25.9437627Z Getting action download info 2025-11-03T16:14:26.0886050Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (3f6538febd727b782e6e13cfd026a309fb14351d) 2025-11-03T16:14:26.0889410Z ##[group] Inputs 2025-11-03T16:14:26.0889757Z build-environment: linux-jammy-py3.10-gcc11-build 2025-11-03T16:14:26.0891379Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-11-03T16:14:26.0893239Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:14:26.0893804Z sync-tag: 2025-11-03T16:14:26.0894480Z timeout-minutes: 240 2025-11-03T16:14:26.0894737Z use-gha: 2025-11-03T16:14:26.0894966Z dashboard-tag: 2025-11-03T16:14:26.0895189Z s3-bucket: gha-artifacts 2025-11-03T16:14:26.0895743Z aws-role-to-assume: 2025-11-03T16:14:26.0896224Z disable-monitor: false 2025-11-03T16:14:26.0896470Z monitor-log-interval: 5 2025-11-03T16:14:26.0896787Z monitor-data-collect-interval: 1 2025-11-03T16:14:26.0897064Z ##[endgroup] 2025-11-03T16:14:26.0897464Z Complete job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:14:26.1359476Z A job started hook has been configured by the self-hosted runner administrator 2025-11-03T16:14:26.1436705Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-11-03T16:14:26.1443830Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:14:26.1444273Z ##[endgroup] 2025-11-03T16:14:27.1472809Z Runner Type: linux.8xlarge.amx 2025-11-03T16:14:27.1473406Z Instance Type: m7i-flex.8xlarge 2025-11-03T16:14:27.1473618Z AMI Name: unknown 2025-11-03T16:14:27.1500751Z AMI ID: ami-08982f1c5bf93d976 2025-11-03T16:14:31.2402516Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-11-03T16:14:31.2402826Z with: 2025-11-03T16:14:31.2403380Z github-secret: *** 2025-11-03T16:14:31.2403817Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-11-03T16:14:31.2404262Z activate-with-label: false 2025-11-03T16:14:31.2404453Z label: with-ssh 2025-11-03T16:14:31.2404614Z remove-existing-keys: true 2025-11-03T16:14:31.2404795Z fail-silently: true 2025-11-03T16:14:31.2404958Z env: 2025-11-03T16:14:31.2405121Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:14:31.2405294Z ##[endgroup] 2025-11-03T16:14:31.3745379Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-11-03T16:14:31.3747886Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-11-03T16:14:31.4138782Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-11-03T16:14:31.4139061Z with: 2025-11-03T16:14:31.4139216Z no-sudo: true 2025-11-03T16:14:31.4139392Z submodules: recursive 2025-11-03T16:14:31.4139572Z fetch-depth: 0 2025-11-03T16:14:31.4139717Z env: 2025-11-03T16:14:31.4139861Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:14:31.4140031Z ##[endgroup] 2025-11-03T16:14:31.4206973Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-11-03T16:14:31.4207556Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-11-03T16:14:31.4214859Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:14:31.4215116Z env: 2025-11-03T16:14:31.4215280Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:14:31.4215457Z ##[endgroup] 2025-11-03T16:14:31.4297977Z ##[group]Run # Use all available CPUs for fetching 2025-11-03T16:14:31.4298279Z # Use all available CPUs for fetching 2025-11-03T16:14:31.4298514Z cd "${GITHUB_WORKSPACE}" 2025-11-03T16:14:31.4298782Z git config --global fetch.parallel 0 2025-11-03T16:14:31.4299040Z git config --global submodule.fetchJobs 0 2025-11-03T16:14:31.4299252Z  2025-11-03T16:14:31.4299483Z # Clean workspace. The default checkout action should also do this, but 2025-11-03T16:14:31.4299758Z # do it here as well just in case 2025-11-03T16:14:31.4299961Z if [[ -d .git ]]; then 2025-11-03T16:14:31.4300155Z  if [ -z "${NO_SUDO}" ]; then 2025-11-03T16:14:31.4300358Z  sudo git clean -ffdx 2025-11-03T16:14:31.4300532Z  else 2025-11-03T16:14:31.4300691Z  git clean -ffdx 2025-11-03T16:14:31.4300863Z  fi 2025-11-03T16:14:31.4301007Z fi 2025-11-03T16:14:31.4304875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:14:31.4305116Z env: 2025-11-03T16:14:31.4305361Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:14:31.4305551Z NO_SUDO: true 2025-11-03T16:14:31.4305706Z ##[endgroup] 2025-11-03T16:14:31.4411571Z ##[group]Run actions/checkout@v4 2025-11-03T16:14:31.4411776Z with: 2025-11-03T16:14:31.4411947Z ref: 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:14:31.4412151Z fetch-depth: 0 2025-11-03T16:14:31.4412317Z submodules: recursive 2025-11-03T16:14:31.4412489Z show-progress: false 2025-11-03T16:14:31.4412665Z repository: pytorch/pytorch 2025-11-03T16:14:31.4412920Z token: *** 2025-11-03T16:14:31.4413072Z ssh-strict: true 2025-11-03T16:14:31.4413436Z ssh-user: git 2025-11-03T16:14:31.4413608Z persist-credentials: true 2025-11-03T16:14:31.4413776Z clean: true 2025-11-03T16:14:31.4413958Z sparse-checkout-cone-mode: true 2025-11-03T16:14:31.4414168Z fetch-tags: false 2025-11-03T16:14:31.4414336Z lfs: false 2025-11-03T16:14:31.4414493Z set-safe-directory: true 2025-11-03T16:14:31.4414696Z env: 2025-11-03T16:14:31.4414858Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:14:31.4415039Z ##[endgroup] 2025-11-03T16:14:31.5367556Z Syncing repository: pytorch/pytorch 2025-11-03T16:14:31.5368574Z ##[group]Getting Git version info 2025-11-03T16:14:31.5368887Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-11-03T16:14:31.5369299Z [command]/usr/bin/git version 2025-11-03T16:14:31.5588387Z git version 2.50.1 2025-11-03T16:14:31.5642377Z ##[endgroup] 2025-11-03T16:14:31.5650440Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/4fda878b-8eed-44dd-8238-dd27ce4d6bca/.gitconfig' 2025-11-03T16:14:31.5683656Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4fda878b-8eed-44dd-8238-dd27ce4d6bca' before making global git config changes 2025-11-03T16:14:31.5688666Z Adding repository directory to the temporary git global config as a safe directory 2025-11-03T16:14:31.5694198Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-11-03T16:14:31.5751080Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-11-03T16:14:31.5759524Z ##[group]Initializing the repository 2025-11-03T16:14:31.5765704Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-11-03T16:14:31.5839279Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-11-03T16:14:31.5844022Z hint: is subject to change. To configure the initial branch name to use in all 2025-11-03T16:14:31.5848407Z hint: of your new repositories, which will suppress this warning, call: 2025-11-03T16:14:31.5852755Z hint: 2025-11-03T16:14:31.5858221Z hint: git config --global init.defaultBranch 2025-11-03T16:14:31.5862800Z hint: 2025-11-03T16:14:31.5867362Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-11-03T16:14:31.5869383Z hint: 'development'. The just-created branch can be renamed via this command: 2025-11-03T16:14:31.5869672Z hint: 2025-11-03T16:14:31.5869847Z hint: git branch -m 2025-11-03T16:14:31.5870029Z hint: 2025-11-03T16:14:31.5870268Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-11-03T16:14:31.5870709Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-11-03T16:14:31.5871466Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-11-03T16:14:31.5908476Z ##[endgroup] 2025-11-03T16:14:31.5909742Z ##[group]Disabling automatic garbage collection 2025-11-03T16:14:31.5910331Z [command]/usr/bin/git config --local gc.auto 0 2025-11-03T16:14:31.5947498Z ##[endgroup] 2025-11-03T16:14:31.5947869Z ##[group]Setting up auth 2025-11-03T16:14:31.5948183Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-11-03T16:14:31.5983320Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-11-03T16:14:31.6313176Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-11-03T16:14:31.6348894Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-11-03T16:14:31.6698306Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-11-03T16:14:31.6760068Z ##[endgroup] 2025-11-03T16:14:31.6764167Z ##[group]Fetching the repository 2025-11-03T16:14:31.6769113Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-11-03T16:15:21.9016319Z From https://github.com/pytorch/pytorch 2025-11-03T16:15:21.9016754Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-11-03T16:15:21.9017482Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-11-03T16:15:21.9018023Z * [new branch] DyVariaSourceRepr -> origin/DyVariaSourceRepr 2025-11-03T16:15:21.9020933Z * [new branch] DynamoFixGit -> origin/DynamoFixGit 2025-11-03T16:15:21.9025601Z * [new branch] DynamoVariaT -> origin/DynamoVariaT 2025-11-03T16:15:21.9031058Z * [new branch] Flamefire-misplaced-TestFailure -> origin/Flamefire-misplaced-TestFailure 2025-11-03T16:15:21.9035814Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-11-03T16:15:21.9040313Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-11-03T16:15:21.9045126Z * [new branch] IvanKobzarev/stack/2 -> origin/IvanKobzarev/stack/2 2025-11-03T16:15:21.9047701Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-11-03T16:15:21.9048263Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-11-03T16:15:21.9054102Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-11-03T16:15:21.9059224Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-11-03T16:15:21.9063368Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-11-03T16:15:21.9067954Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-11-03T16:15:21.9070232Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-11-03T16:15:21.9070746Z * [new branch] VLA_exp -> origin/VLA_exp 2025-11-03T16:15:21.9075234Z * [new branch] VariaT -> origin/VariaT 2025-11-03T16:15:21.9078362Z * [new branch] add-pyrefly-to-lintrunner -> origin/add-pyrefly-to-lintrunner 2025-11-03T16:15:21.9083173Z * [new branch] add_conv3d_ops -> origin/add_conv3d_ops 2025-11-03T16:15:21.9088729Z * [new branch] add_op_to_dashboard -> origin/add_op_to_dashboard 2025-11-03T16:15:21.9092618Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-11-03T16:15:21.9097271Z * [new branch] addmm10000rtx -> origin/addmm10000rtx 2025-11-03T16:15:21.9102756Z * [new branch] adi/onednn_aarch64 -> origin/adi/onednn_aarch64 2025-11-03T16:15:21.9107153Z * [new branch] adi/test -> origin/adi/test 2025-11-03T16:15:21.9111968Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-11-03T16:15:21.9112433Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-11-03T16:15:21.9112774Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-11-03T16:15:21.9113350Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-11-03T16:15:21.9114122Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-11-03T16:15:21.9114494Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-11-03T16:15:21.9114846Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-11-03T16:15:21.9115194Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-11-03T16:15:21.9115591Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-11-03T16:15:21.9115899Z * [new branch] alt-disable -> origin/alt-disable 2025-11-03T16:15:21.9116198Z * [new branch] angelayi/allow_fake -> origin/angelayi/allow_fake 2025-11-03T16:15:21.9116550Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-11-03T16:15:21.9116898Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-11-03T16:15:21.9117264Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-11-03T16:15:21.9117625Z * [new branch] angelayi/const_folder -> origin/angelayi/const_folder 2025-11-03T16:15:21.9117939Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-11-03T16:15:21.9118248Z * [new branch] angelayi/joint_kwargs -> origin/angelayi/joint_kwargs 2025-11-03T16:15:21.9118558Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-11-03T16:15:21.9118872Z * [new branch] angelayi/opaque_obj_v2 -> origin/angelayi/opaque_obj_v2 2025-11-03T16:15:21.9119246Z * [new branch] angelayi/post_grad -> origin/angelayi/post_grad 2025-11-03T16:15:21.9119563Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-11-03T16:15:21.9119874Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-11-03T16:15:21.9120197Z * [new branch] angelayi/state_dict -> origin/angelayi/state_dict 2025-11-03T16:15:21.9120541Z * [new branch] angelayi/static_input_indices -> origin/angelayi/static_input_indices 2025-11-03T16:15:21.9120883Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-11-03T16:15:21.9121201Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-11-03T16:15:21.9121513Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-11-03T16:15:21.9121814Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-11-03T16:15:21.9122146Z * [new branch] annotate_before_eliminate -> origin/annotate_before_eliminate 2025-11-03T16:15:21.9122455Z * [new branch] annotate_fn -> origin/annotate_fn 2025-11-03T16:15:21.9122742Z * [new branch] annotation_bw -> origin/annotation_bw 2025-11-03T16:15:21.9123038Z * [new branch] annotation_bw_1 -> origin/annotation_bw_1 2025-11-03T16:15:21.9123342Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-11-03T16:15:21.9123636Z * [new branch] annotation_log -> origin/annotation_log 2025-11-03T16:15:21.9123970Z * [new branch] annotation_replay -> origin/annotation_replay 2025-11-03T16:15:21.9124268Z * [new branch] annotation_submod -> origin/annotation_submod 2025-11-03T16:15:21.9124630Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-11-03T16:15:21.9124946Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-11-03T16:15:21.9125263Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-11-03T16:15:21.9125655Z * [new branch] aoti_metal_shimify -> origin/aoti_metal_shimify 2025-11-03T16:15:21.9126061Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-11-03T16:15:21.9126390Z * [new branch] aoti_shim_library_list -> origin/aoti_shim_library_list 2025-11-03T16:15:21.9126707Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-11-03T16:15:21.9127008Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-11-03T16:15:21.9127367Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-11-03T16:15:21.9127716Z * [new branch] async_tp -> origin/async_tp 2025-11-03T16:15:21.9128041Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-11-03T16:15:21.9128440Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-11-03T16:15:21.9128799Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-11-03T16:15:21.9129101Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-11-03T16:15:21.9129396Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-11-03T16:15:21.9129678Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-11-03T16:15:21.9129985Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-11-03T16:15:21.9130302Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-11-03T16:15:21.9130656Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-11-03T16:15:21.9130963Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-11-03T16:15:21.9131285Z * [new branch] attention_benchmark -> origin/attention_benchmark 2025-11-03T16:15:21.9131628Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-11-03T16:15:21.9131980Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-11-03T16:15:21.9132308Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-11-03T16:15:21.9132616Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-11-03T16:15:21.9132918Z * [new branch] bahuang/test -> origin/bahuang/test 2025-11-03T16:15:21.9133200Z * [new branch] base/1.5 -> origin/base/1.5 2025-11-03T16:15:21.9133536Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-11-03T16:15:21.9133899Z * [new branch] bench_scaled_mm_ops -> origin/bench_scaled_mm_ops 2025-11-03T16:15:21.9134207Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-11-03T16:15:21.9134533Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-11-03T16:15:21.9134861Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-11-03T16:15:21.9135176Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-11-03T16:15:21.9135476Z * [new branch] bf/cg-backend -> origin/bf/cg-backend 2025-11-03T16:15:21.9135840Z * [new branch] bf/cg-partition-custom-op-mutation -> origin/bf/cg-partition-custom-op-mutation 2025-11-03T16:15:21.9136222Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-11-03T16:15:21.9136526Z * [new branch] bf/clean-hf -> origin/bf/clean-hf 2025-11-03T16:15:21.9136815Z * [new branch] bf/clean-timm -> origin/bf/clean-timm 2025-11-03T16:15:21.9137159Z * [new branch] bf/clean-torchbench -> origin/bf/clean-torchbench 2025-11-03T16:15:21.9137462Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-11-03T16:15:21.9137764Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-11-03T16:15:21.9138126Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-11-03T16:15:21.9138670Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-11-03T16:15:21.9139157Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-11-03T16:15:21.9139484Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-11-03T16:15:21.9139807Z * [new branch] bf/dynamo-partition -> origin/bf/dynamo-partition 2025-11-03T16:15:21.9140125Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-11-03T16:15:21.9140477Z * [new branch] bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols 2025-11-03T16:15:21.9140852Z * [new branch] bf/partition-custom-rules -> origin/bf/partition-custom-rules 2025-11-03T16:15:21.9141199Z * [new branch] bf/partition-memory-plan -> origin/bf/partition-memory-plan 2025-11-03T16:15:21.9141546Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-11-03T16:15:21.9141891Z * [new branch] bf/partition-view-fallback -> origin/bf/partition-view-fallback 2025-11-03T16:15:21.9142279Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-11-03T16:15:21.9142623Z * [new branch] bf16_support_per_channel -> origin/bf16_support_per_channel 2025-11-03T16:15:21.9142957Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-11-03T16:15:21.9143313Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-11-03T16:15:21.9143653Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-11-03T16:15:21.9143993Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-11-03T16:15:21.9144330Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-11-03T16:15:21.9144659Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-11-03T16:15:21.9145004Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-11-03T16:15:21.9145346Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-11-03T16:15:21.9145687Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-11-03T16:15:21.9146056Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-11-03T16:15:21.9146393Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-11-03T16:15:21.9146709Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-11-03T16:15:21.9147051Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-11-03T16:15:21.9147400Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-11-03T16:15:21.9147742Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-11-03T16:15:21.9148079Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-11-03T16:15:21.9148387Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-11-03T16:15:21.9148765Z * [new branch] brister/break_dynamic_scalar -> origin/brister/break_dynamic_scalar 2025-11-03T16:15:21.9149111Z * [new branch] brister/cast_rank0 -> origin/brister/cast_rank0 2025-11-03T16:15:21.9149424Z * [new branch] brister/item_fx -> origin/brister/item_fx 2025-11-03T16:15:21.9149766Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-11-03T16:15:21.9150161Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-11-03T16:15:21.9150536Z * [new branch] bwd-backup -> origin/bwd-backup 2025-11-03T16:15:21.9150830Z * [new branch] c57382a49 -> origin/c57382a49 2025-11-03T16:15:21.9151117Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-11-03T16:15:21.9151418Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-11-03T16:15:21.9151757Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-11-03T16:15:21.9152195Z * [new branch] cherry-pick-152361-by-pytorch_bot_bot_ -> origin/cherry-pick-152361-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9152671Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9153156Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9153629Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9154274Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9154786Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9155295Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9155796Z * [new branch] cherry-pick-162194-by-pytorch_bot_bot_ -> origin/cherry-pick-162194-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9156275Z * [new branch] cherry-pick-162693-by-pytorch_bot_bot_ -> origin/cherry-pick-162693-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9156730Z * [new branch] cherry-pick-162764-by-pytorch_bot_bot_ -> origin/cherry-pick-162764-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9157194Z * [new branch] cherry-pick-163029-by-pytorch_bot_bot_ -> origin/cherry-pick-163029-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9157661Z * [new branch] cherry-pick-163776-by-pytorch_bot_bot_ -> origin/cherry-pick-163776-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9158121Z * [new branch] cherry-pick-164774-by-pytorch_bot_bot_ -> origin/cherry-pick-164774-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9158581Z * [new branch] cherry-pick-164870-by-pytorch_bot_bot_ -> origin/cherry-pick-164870-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9159037Z * [new branch] cherry-pick-164946-by-pytorch_bot_bot_ -> origin/cherry-pick-164946-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9159486Z * [new branch] cherry-pick-165013-by-pytorch_bot_bot_ -> origin/cherry-pick-165013-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9159946Z * [new branch] cherry-pick-165465-by-pytorch_bot_bot_ -> origin/cherry-pick-165465-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9160445Z * [new branch] cherry-pick-165665-by-pytorch_bot_bot_ -> origin/cherry-pick-165665-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9160905Z * [new branch] cherry-pick-165794-by-pytorch_bot_bot_ -> origin/cherry-pick-165794-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9161366Z * [new branch] cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_ 2025-11-03T16:15:21.9161836Z * [new branch] chuanqi129-patch-1 -> origin/chuanqi129-patch-1 2025-11-03T16:15:21.9162146Z * [new branch] ci_attn -> origin/ci_attn 2025-11-03T16:15:21.9162487Z * [new branch] codegen_trace -> origin/codegen_trace 2025-11-03T16:15:21.9162835Z * [new branch] codex-testing -> origin/codex-testing 2025-11-03T16:15:21.9163282Z * [new branch] codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions 2025-11-03T16:15:21.9163922Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-11-03T16:15:21.9164582Z * [new branch] codex/enhance-cuda.matmul-with-allow_splitk-argument -> origin/codex/enhance-cuda.matmul-with-allow_splitk-argument 2025-11-03T16:15:21.9165118Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-11-03T16:15:21.9165639Z * [new branch] codex/refactor-dimension-handling-in-shape.cu -> origin/codex/refactor-dimension-handling-in-shape.cu 2025-11-03T16:15:21.9166243Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-11-03T16:15:21.9166698Z * [new branch] context_test -> origin/context_test 2025-11-03T16:15:21.9167095Z * [new branch] copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip 2025-11-03T16:15:21.9167523Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-11-03T16:15:21.9167953Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-11-03T16:15:21.9168322Z * [new branch] crcrpar-patch-1 -> origin/crcrpar-patch-1 2025-11-03T16:15:21.9168654Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-11-03T16:15:21.9168996Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-11-03T16:15:21.9169343Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-11-03T16:15:21.9169705Z * [new branch] csl/debug_tests_larger_runner -> origin/csl/debug_tests_larger_runner 2025-11-03T16:15:21.9170095Z * [new branch] csl/disable_test_leaking_memory -> origin/csl/disable_test_leaking_memory 2025-11-03T16:15:21.9170470Z * [new branch] csl/failed_consistently_null -> origin/csl/failed_consistently_null 2025-11-03T16:15:21.9170840Z * [new branch] csl/fix_internal_graph_executor -> origin/csl/fix_internal_graph_executor 2025-11-03T16:15:21.9171209Z * [new branch] csl/fix_periodic_debug_1 -> origin/csl/fix_periodic_debug_1 2025-11-03T16:15:21.9171578Z * [new branch] csl/inductor_h100_nightly -> origin/csl/inductor_h100_nightly 2025-11-03T16:15:21.9171906Z * [new branch] csl/katex -> origin/csl/katex 2025-11-03T16:15:21.9172228Z * [new branch] csl/keep_going_manual -> origin/csl/keep_going_manual 2025-11-03T16:15:21.9172534Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-11-03T16:15:21.9172839Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-11-03T16:15:21.9173137Z * [new branch] csl/lint_thing -> origin/csl/lint_thing 2025-11-03T16:15:21.9173451Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-11-03T16:15:21.9173807Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-11-03T16:15:21.9174125Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-11-03T16:15:21.9174498Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-11-03T16:15:21.9174858Z * [new branch] csl/nogpu_inductor_caching_test -> origin/csl/nogpu_inductor_caching_test 2025-11-03T16:15:21.9175215Z * [new branch] csl/nogpu_on_no_gpu -> origin/csl/nogpu_on_no_gpu 2025-11-03T16:15:21.9175555Z * [new branch] csl/periodic_disable -> origin/csl/periodic_disable 2025-11-03T16:15:21.9175948Z * [new branch] csl/print_something_for_log_classifier -> origin/csl/print_something_for_log_classifier 2025-11-03T16:15:21.9176335Z * [new branch] csl/print_timing -> origin/csl/print_timing 2025-11-03T16:15:21.9176679Z * [new branch] csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var 2025-11-03T16:15:21.9177038Z * [new branch] csl/remove_run_parallel -> origin/csl/remove_run_parallel 2025-11-03T16:15:21.9177378Z * [new branch] csl/remove_unused_vars -> origin/csl/remove_unused_vars 2025-11-03T16:15:21.9177709Z * [new branch] csl/revert_no_rule -> origin/csl/revert_no_rule 2025-11-03T16:15:21.9178030Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-11-03T16:15:21.9178345Z * [new branch] csl/run_full_lint_trunk -> origin/csl/run_full_lint_trunk 2025-11-03T16:15:21.9178711Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-11-03T16:15:21.9179049Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-11-03T16:15:21.9179391Z * [new branch] csl/td_job_level -> origin/csl/td_job_level 2025-11-03T16:15:21.9179779Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-11-03T16:15:21.9180198Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-11-03T16:15:21.9180620Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-11-03T16:15:21.9180996Z * [new branch] csl/upload_all_data -> origin/csl/upload_all_data 2025-11-03T16:15:21.9181326Z * [new branch] csl/win_cpp_tests -> origin/csl/win_cpp_tests 2025-11-03T16:15:21.9181647Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-11-03T16:15:21.9182007Z * [new branch] csl/workflow_consistency_linter -> origin/csl/workflow_consistency_linter 2025-11-03T16:15:21.9182389Z * [new branch] csl/workflow_sync_more_files -> origin/csl/workflow_sync_more_files 2025-11-03T16:15:21.9182727Z * [new branch] csl/xml_stuff -> origin/csl/xml_stuff 2025-11-03T16:15:21.9183031Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-11-03T16:15:21.9183342Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-11-03T16:15:21.9183665Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-11-03T16:15:21.9183975Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-11-03T16:15:21.9184268Z * [new branch] debug-guard -> origin/debug-guard 2025-11-03T16:15:21.9184575Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-11-03T16:15:21.9185124Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-11-03T16:15:21.9185850Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 2025-11-03T16:15:21.9186425Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-11-03T16:15:21.9186885Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-11-03T16:15:21.9187292Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-11-03T16:15:21.9187662Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-11-03T16:15:21.9188009Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-11-03T16:15:21.9188317Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-11-03T16:15:21.9188678Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-11-03T16:15:21.9189046Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-11-03T16:15:21.9189435Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-11-03T16:15:21.9189772Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-11-03T16:15:21.9190084Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-11-03T16:15:21.9190407Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-11-03T16:15:21.9190714Z * [new branch] disp_counter -> origin/disp_counter 2025-11-03T16:15:21.9191005Z * [new branch] document -> origin/document 2025-11-03T16:15:21.9191299Z * [new branch] document-apis -> origin/document-apis 2025-11-03T16:15:21.9191602Z * [new branch] dynamo_stack_trace -> origin/dynamo_stack_trace 2025-11-03T16:15:21.9191977Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-11-03T16:15:21.9192330Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-11-03T16:15:21.9192697Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-11-03T16:15:21.9193063Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-11-03T16:15:21.9193437Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-11-03T16:15:21.9193802Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-11-03T16:15:21.9194256Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-11-03T16:15:21.9194664Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-11-03T16:15:21.9195111Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-11-03T16:15:21.9195529Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-11-03T16:15:21.9196001Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-11-03T16:15:21.9196499Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-11-03T16:15:21.9197009Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-11-03T16:15:21.9197474Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-11-03T16:15:21.9198361Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-11-03T16:15:21.9198899Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-11-03T16:15:21.9199381Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-11-03T16:15:21.9200043Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-11-03T16:15:21.9200901Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-11-03T16:15:21.9201489Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-11-03T16:15:21.9202159Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-11-03T16:15:21.9202690Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-11-03T16:15:21.9203185Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-11-03T16:15:21.9203686Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-11-03T16:15:21.9204179Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-11-03T16:15:21.9204801Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-11-03T16:15:21.9206805Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-11-03T16:15:21.9207186Z * [new branch] exec -> origin/exec 2025-11-03T16:15:21.9207507Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-11-03T16:15:21.9207889Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-11-03T16:15:21.9208595Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-11-03T16:15:21.9209416Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-11-03T16:15:21.9210253Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-11-03T16:15:21.9210706Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-11-03T16:15:21.9211330Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-11-03T16:15:21.9211996Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-11-03T16:15:21.9212587Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-11-03T16:15:21.9213363Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-11-03T16:15:21.9213780Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-11-03T16:15:21.9214676Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-11-03T16:15:21.9215336Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-11-03T16:15:21.9215818Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-11-03T16:15:21.9216443Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-11-03T16:15:21.9217116Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-11-03T16:15:21.9217712Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-11-03T16:15:21.9218297Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-11-03T16:15:21.9218926Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-11-03T16:15:21.9219541Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-11-03T16:15:21.9220162Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-11-03T16:15:21.9220763Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-11-03T16:15:21.9221347Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-11-03T16:15:21.9221994Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-11-03T16:15:21.9222756Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-11-03T16:15:21.9223417Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-11-03T16:15:21.9224031Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-11-03T16:15:21.9224654Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-11-03T16:15:21.9225298Z * [new branch] export-D81698719 -> origin/export-D81698719 2025-11-03T16:15:21.9226366Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-11-03T16:15:21.9226834Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-11-03T16:15:21.9227552Z * [new branch] export-D83390563 -> origin/export-D83390563 2025-11-03T16:15:21.9228168Z * [new branch] export-D83391942 -> origin/export-D83391942 2025-11-03T16:15:21.9228837Z * [new branch] export-D83395610 -> origin/export-D83395610 2025-11-03T16:15:21.9229468Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-11-03T16:15:21.9230539Z * [new branch] export-D83591083 -> origin/export-D83591083 2025-11-03T16:15:21.9231150Z * [new branch] export-D83609850 -> origin/export-D83609850 2025-11-03T16:15:21.9231782Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-11-03T16:15:21.9232446Z * [new branch] export-D83714690 -> origin/export-D83714690 2025-11-03T16:15:21.9233157Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-11-03T16:15:21.9233858Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-11-03T16:15:21.9234513Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-11-03T16:15:21.9235338Z * [new branch] export-D84009392 -> origin/export-D84009392 2025-11-03T16:15:21.9236059Z * [new branch] export-D84025878 -> origin/export-D84025878 2025-11-03T16:15:21.9236697Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-11-03T16:15:21.9237360Z * [new branch] export-D84098898 -> origin/export-D84098898 2025-11-03T16:15:21.9238043Z * [new branch] export-D84103213 -> origin/export-D84103213 2025-11-03T16:15:21.9239269Z * [new branch] export-D84169910 -> origin/export-D84169910 2025-11-03T16:15:21.9239609Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-11-03T16:15:21.9241997Z * [new branch] export-D84373821 -> origin/export-D84373821 2025-11-03T16:15:21.9242455Z * [new branch] export-D84522373 -> origin/export-D84522373 2025-11-03T16:15:21.9242824Z * [new branch] export-D84612194 -> origin/export-D84612194 2025-11-03T16:15:21.9243189Z * [new branch] export-D84890985 -> origin/export-D84890985 2025-11-03T16:15:21.9243591Z * [new branch] export-D85122326 -> origin/export-D85122326 2025-11-03T16:15:21.9244280Z * [new branch] export-D85745809 -> origin/export-D85745809 2025-11-03T16:15:21.9245026Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-11-03T16:15:21.9245709Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-11-03T16:15:21.9246354Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-11-03T16:15:21.9246958Z * [new branch] ezyang-war -> origin/ezyang-war 2025-11-03T16:15:21.9248142Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-11-03T16:15:21.9248728Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-11-03T16:15:21.9249398Z * [new branch] fadeputr-fix-fbgemm_genai-build -> origin/fadeputr-fix-fbgemm_genai-build 2025-11-03T16:15:21.9250398Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-11-03T16:15:21.9250952Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-11-03T16:15:21.9254997Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-11-03T16:15:21.9255339Z * [new branch] fca -> origin/fca 2025-11-03T16:15:21.9255633Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-11-03T16:15:21.9255924Z * [new branch] fca5 -> origin/fca5 2025-11-03T16:15:21.9256239Z * [new branch] feature/forkserver-numa -> origin/feature/forkserver-numa 2025-11-03T16:15:21.9256780Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-11-03T16:15:21.9257152Z * [new branch] feature/numa-forkserver -> origin/feature/numa-forkserver 2025-11-03T16:15:21.9257626Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-11-03T16:15:21.9258297Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-11-03T16:15:21.9261025Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-11-03T16:15:21.9261437Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-11-03T16:15:21.9261781Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-11-03T16:15:21.9262354Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-11-03T16:15:21.9262714Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-11-03T16:15:21.9263086Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-11-03T16:15:21.9263452Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-11-03T16:15:21.9263805Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-11-03T16:15:21.9264711Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-11-03T16:15:21.9265302Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-11-03T16:15:21.9266069Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-11-03T16:15:21.9266786Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-11-03T16:15:21.9267494Z * [new branch] fix_addmm_issue -> origin/fix_addmm_issue 2025-11-03T16:15:21.9268272Z * [new branch] fix_fx_graph_print_space -> origin/fix_fx_graph_print_space 2025-11-03T16:15:21.9268899Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-11-03T16:15:21.9269534Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-11-03T16:15:21.9270184Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-11-03T16:15:21.9270839Z * [new branch] fixbugh100 -> origin/fixbugh100 2025-11-03T16:15:21.9271563Z * [new branch] fixes-triage -> origin/fixes-triage 2025-11-03T16:15:21.9272242Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-11-03T16:15:21.9272876Z * [new branch] fixvllmoct -> origin/fixvllmoct 2025-11-03T16:15:21.9273642Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-11-03T16:15:21.9274280Z * [new branch] flex-flash -> origin/flex-flash 2025-11-03T16:15:21.9275162Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-11-03T16:15:21.9275854Z * [new branch] flex_flash -> origin/flex_flash 2025-11-03T16:15:21.9277653Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-11-03T16:15:21.9278060Z * [new branch] forkserver_fix -> origin/forkserver_fix 2025-11-03T16:15:21.9278651Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-11-03T16:15:21.9278998Z * [new branch] fx_cpp -> origin/fx_cpp 2025-11-03T16:15:21.9282218Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-11-03T16:15:21.9282634Z * [new branch] georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch 2025-11-03T16:15:21.9289024Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-11-03T16:15:21.9290895Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-11-03T16:15:21.9291318Z * [new branch] gh/ColinPeppler/94/base -> origin/gh/ColinPeppler/94/base 2025-11-03T16:15:21.9291733Z * [new branch] gh/ColinPeppler/94/head -> origin/gh/ColinPeppler/94/head 2025-11-03T16:15:21.9292127Z * [new branch] gh/ColinPeppler/94/orig -> origin/gh/ColinPeppler/94/orig 2025-11-03T16:15:21.9292506Z * [new branch] gh/ColinPeppler/95/base -> origin/gh/ColinPeppler/95/base 2025-11-03T16:15:21.9292896Z * [new branch] gh/ColinPeppler/95/head -> origin/gh/ColinPeppler/95/head 2025-11-03T16:15:21.9293429Z * [new branch] gh/ColinPeppler/95/orig -> origin/gh/ColinPeppler/95/orig 2025-11-03T16:15:21.9293838Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-11-03T16:15:21.9294150Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-11-03T16:15:21.9295424Z * [new branch] gh/EikanWang/68/base -> origin/gh/EikanWang/68/base 2025-11-03T16:15:21.9295819Z * [new branch] gh/EikanWang/68/head -> origin/gh/EikanWang/68/head 2025-11-03T16:15:21.9301547Z * [new branch] gh/EikanWang/68/orig -> origin/gh/EikanWang/68/orig 2025-11-03T16:15:21.9305548Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-11-03T16:15:21.9307292Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-11-03T16:15:21.9307637Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-11-03T16:15:21.9307972Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-11-03T16:15:21.9308278Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-11-03T16:15:21.9308595Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-11-03T16:15:21.9308883Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-11-03T16:15:21.9309181Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-11-03T16:15:21.9309478Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-11-03T16:15:21.9309775Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-11-03T16:15:21.9310075Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-11-03T16:15:21.9310369Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-11-03T16:15:21.9310673Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-11-03T16:15:21.9310974Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-11-03T16:15:21.9311408Z * [new branch] gh/H-Huang/212/base -> origin/gh/H-Huang/212/base 2025-11-03T16:15:21.9311699Z * [new branch] gh/H-Huang/212/head -> origin/gh/H-Huang/212/head 2025-11-03T16:15:21.9311998Z * [new branch] gh/H-Huang/212/orig -> origin/gh/H-Huang/212/orig 2025-11-03T16:15:21.9312296Z * [new branch] gh/H-Huang/214/base -> origin/gh/H-Huang/214/base 2025-11-03T16:15:21.9312597Z * [new branch] gh/H-Huang/214/head -> origin/gh/H-Huang/214/head 2025-11-03T16:15:21.9312917Z * [new branch] gh/H-Huang/214/orig -> origin/gh/H-Huang/214/orig 2025-11-03T16:15:21.9313364Z * [new branch] gh/H-Huang/216/base -> origin/gh/H-Huang/216/base 2025-11-03T16:15:21.9313686Z * [new branch] gh/H-Huang/216/head -> origin/gh/H-Huang/216/head 2025-11-03T16:15:21.9314142Z * [new branch] gh/H-Huang/216/orig -> origin/gh/H-Huang/216/orig 2025-11-03T16:15:21.9314497Z * [new branch] gh/H-Huang/218/base -> origin/gh/H-Huang/218/base 2025-11-03T16:15:21.9314841Z * [new branch] gh/H-Huang/218/head -> origin/gh/H-Huang/218/head 2025-11-03T16:15:21.9315183Z * [new branch] gh/H-Huang/218/orig -> origin/gh/H-Huang/218/orig 2025-11-03T16:15:21.9315491Z * [new branch] gh/H-Huang/219/base -> origin/gh/H-Huang/219/base 2025-11-03T16:15:21.9315797Z * [new branch] gh/H-Huang/219/head -> origin/gh/H-Huang/219/head 2025-11-03T16:15:21.9316086Z * [new branch] gh/H-Huang/219/orig -> origin/gh/H-Huang/219/orig 2025-11-03T16:15:21.9316493Z * [new branch] gh/H-Huang/220/base -> origin/gh/H-Huang/220/base 2025-11-03T16:15:21.9316789Z * [new branch] gh/H-Huang/220/head -> origin/gh/H-Huang/220/head 2025-11-03T16:15:21.9317279Z * [new branch] gh/H-Huang/220/orig -> origin/gh/H-Huang/220/orig 2025-11-03T16:15:21.9318343Z * [new branch] gh/H-Huang/221/base -> origin/gh/H-Huang/221/base 2025-11-03T16:15:21.9318800Z * [new branch] gh/H-Huang/221/head -> origin/gh/H-Huang/221/head 2025-11-03T16:15:21.9319447Z * [new branch] gh/H-Huang/221/orig -> origin/gh/H-Huang/221/orig 2025-11-03T16:15:21.9320575Z * [new branch] gh/H-Huang/222/base -> origin/gh/H-Huang/222/base 2025-11-03T16:15:21.9320887Z * [new branch] gh/H-Huang/222/head -> origin/gh/H-Huang/222/head 2025-11-03T16:15:21.9321508Z * [new branch] gh/H-Huang/222/orig -> origin/gh/H-Huang/222/orig 2025-11-03T16:15:21.9323046Z * [new branch] gh/H-Huang/223/base -> origin/gh/H-Huang/223/base 2025-11-03T16:15:21.9323359Z * [new branch] gh/H-Huang/223/head -> origin/gh/H-Huang/223/head 2025-11-03T16:15:21.9323680Z * [new branch] gh/H-Huang/223/orig -> origin/gh/H-Huang/223/orig 2025-11-03T16:15:21.9327726Z * [new branch] gh/H-Huang/224/base -> origin/gh/H-Huang/224/base 2025-11-03T16:15:21.9328093Z * [new branch] gh/H-Huang/224/head -> origin/gh/H-Huang/224/head 2025-11-03T16:15:21.9328403Z * [new branch] gh/H-Huang/224/orig -> origin/gh/H-Huang/224/orig 2025-11-03T16:15:21.9328701Z * [new branch] gh/H-Huang/225/base -> origin/gh/H-Huang/225/base 2025-11-03T16:15:21.9329004Z * [new branch] gh/H-Huang/225/head -> origin/gh/H-Huang/225/head 2025-11-03T16:15:21.9329296Z * [new branch] gh/H-Huang/225/orig -> origin/gh/H-Huang/225/orig 2025-11-03T16:15:21.9329597Z * [new branch] gh/H-Huang/226/base -> origin/gh/H-Huang/226/base 2025-11-03T16:15:21.9329902Z * [new branch] gh/H-Huang/226/head -> origin/gh/H-Huang/226/head 2025-11-03T16:15:21.9330207Z * [new branch] gh/H-Huang/226/orig -> origin/gh/H-Huang/226/orig 2025-11-03T16:15:21.9331390Z * [new branch] gh/H-Huang/227/base -> origin/gh/H-Huang/227/base 2025-11-03T16:15:21.9331768Z * [new branch] gh/H-Huang/227/head -> origin/gh/H-Huang/227/head 2025-11-03T16:15:21.9332371Z * [new branch] gh/H-Huang/227/orig -> origin/gh/H-Huang/227/orig 2025-11-03T16:15:21.9334082Z * [new branch] gh/H-Huang/228/base -> origin/gh/H-Huang/228/base 2025-11-03T16:15:21.9334612Z * [new branch] gh/H-Huang/228/head -> origin/gh/H-Huang/228/head 2025-11-03T16:15:21.9335057Z * [new branch] gh/H-Huang/228/orig -> origin/gh/H-Huang/228/orig 2025-11-03T16:15:21.9335695Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-11-03T16:15:21.9336373Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-11-03T16:15:21.9336987Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-11-03T16:15:21.9339862Z * [new branch] gh/IvanKobzarev/151/base -> origin/gh/IvanKobzarev/151/base 2025-11-03T16:15:21.9340424Z * [new branch] gh/IvanKobzarev/151/head -> origin/gh/IvanKobzarev/151/head 2025-11-03T16:15:21.9340963Z * [new branch] gh/IvanKobzarev/151/orig -> origin/gh/IvanKobzarev/151/orig 2025-11-03T16:15:21.9341409Z * [new branch] gh/IvanKobzarev/156/base -> origin/gh/IvanKobzarev/156/base 2025-11-03T16:15:21.9341742Z * [new branch] gh/IvanKobzarev/156/head -> origin/gh/IvanKobzarev/156/head 2025-11-03T16:15:21.9342223Z * [new branch] gh/IvanKobzarev/156/orig -> origin/gh/IvanKobzarev/156/orig 2025-11-03T16:15:21.9342627Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-11-03T16:15:21.9343146Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-11-03T16:15:21.9343837Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-11-03T16:15:21.9347310Z * [new branch] gh/IvanKobzarev/158/base -> origin/gh/IvanKobzarev/158/base 2025-11-03T16:15:21.9347878Z * [new branch] gh/IvanKobzarev/158/head -> origin/gh/IvanKobzarev/158/head 2025-11-03T16:15:21.9348352Z * [new branch] gh/IvanKobzarev/158/orig -> origin/gh/IvanKobzarev/158/orig 2025-11-03T16:15:21.9348821Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-11-03T16:15:21.9349293Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-11-03T16:15:21.9349661Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-11-03T16:15:21.9350014Z * [new branch] gh/IvanKobzarev/161/base -> origin/gh/IvanKobzarev/161/base 2025-11-03T16:15:21.9350364Z * [new branch] gh/IvanKobzarev/161/head -> origin/gh/IvanKobzarev/161/head 2025-11-03T16:15:21.9350843Z * [new branch] gh/IvanKobzarev/161/orig -> origin/gh/IvanKobzarev/161/orig 2025-11-03T16:15:21.9351196Z * [new branch] gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base 2025-11-03T16:15:21.9352041Z * [new branch] gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head 2025-11-03T16:15:21.9352661Z * [new branch] gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig 2025-11-03T16:15:21.9353824Z * [new branch] gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base 2025-11-03T16:15:21.9354653Z * [new branch] gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head 2025-11-03T16:15:21.9355144Z * [new branch] gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig 2025-11-03T16:15:21.9358044Z * [new branch] gh/IvanKobzarev/164/base -> origin/gh/IvanKobzarev/164/base 2025-11-03T16:15:21.9363232Z * [new branch] gh/IvanKobzarev/164/head -> origin/gh/IvanKobzarev/164/head 2025-11-03T16:15:21.9367937Z * [new branch] gh/IvanKobzarev/164/orig -> origin/gh/IvanKobzarev/164/orig 2025-11-03T16:15:21.9372801Z * [new branch] gh/IvanKobzarev/165/base -> origin/gh/IvanKobzarev/165/base 2025-11-03T16:15:21.9377575Z * [new branch] gh/IvanKobzarev/165/head -> origin/gh/IvanKobzarev/165/head 2025-11-03T16:15:21.9382303Z * [new branch] gh/IvanKobzarev/165/orig -> origin/gh/IvanKobzarev/165/orig 2025-11-03T16:15:21.9387712Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-11-03T16:15:21.9390054Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-11-03T16:15:21.9390451Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-11-03T16:15:21.9390812Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-11-03T16:15:21.9391158Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-11-03T16:15:21.9391519Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-11-03T16:15:21.9391870Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-11-03T16:15:21.9392194Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-11-03T16:15:21.9392506Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-11-03T16:15:21.9392806Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-11-03T16:15:21.9393274Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-11-03T16:15:21.9393585Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-11-03T16:15:21.9393966Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-11-03T16:15:21.9394279Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-11-03T16:15:21.9394574Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-11-03T16:15:21.9394884Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-11-03T16:15:21.9395187Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-11-03T16:15:21.9395492Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-11-03T16:15:21.9395861Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-11-03T16:15:21.9396166Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-11-03T16:15:21.9396463Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-11-03T16:15:21.9396764Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-11-03T16:15:21.9397062Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-11-03T16:15:21.9397351Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-11-03T16:15:21.9397650Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-11-03T16:15:21.9397948Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-11-03T16:15:21.9398245Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-11-03T16:15:21.9398539Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-11-03T16:15:21.9398831Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-11-03T16:15:21.9399126Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-11-03T16:15:21.9399466Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-11-03T16:15:21.9399759Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-11-03T16:15:21.9400054Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-11-03T16:15:21.9400344Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-11-03T16:15:21.9400636Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-11-03T16:15:21.9400930Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-11-03T16:15:21.9401227Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-11-03T16:15:21.9401521Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-11-03T16:15:21.9401818Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-11-03T16:15:21.9402118Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-11-03T16:15:21.9402412Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-11-03T16:15:21.9402716Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-11-03T16:15:21.9403001Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-11-03T16:15:21.9403325Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-11-03T16:15:21.9403662Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-11-03T16:15:21.9403995Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-11-03T16:15:21.9404377Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-11-03T16:15:21.9404707Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-11-03T16:15:21.9405042Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-11-03T16:15:21.9405359Z * [new branch] gh/PaulZhang12/26/base -> origin/gh/PaulZhang12/26/base 2025-11-03T16:15:21.9405675Z * [new branch] gh/PaulZhang12/26/head -> origin/gh/PaulZhang12/26/head 2025-11-03T16:15:21.9405992Z * [new branch] gh/PaulZhang12/26/orig -> origin/gh/PaulZhang12/26/orig 2025-11-03T16:15:21.9406296Z * [new branch] gh/PaulZhang12/27/base -> origin/gh/PaulZhang12/27/base 2025-11-03T16:15:21.9406611Z * [new branch] gh/PaulZhang12/27/head -> origin/gh/PaulZhang12/27/head 2025-11-03T16:15:21.9406928Z * [new branch] gh/PaulZhang12/27/orig -> origin/gh/PaulZhang12/27/orig 2025-11-03T16:15:21.9407239Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-11-03T16:15:21.9407560Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-11-03T16:15:21.9407878Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-11-03T16:15:21.9408197Z * [new branch] gh/PaulZhang12/30/base -> origin/gh/PaulZhang12/30/base 2025-11-03T16:15:21.9408509Z * [new branch] gh/PaulZhang12/30/head -> origin/gh/PaulZhang12/30/head 2025-11-03T16:15:21.9408820Z * [new branch] gh/PaulZhang12/30/orig -> origin/gh/PaulZhang12/30/orig 2025-11-03T16:15:21.9409132Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-11-03T16:15:21.9409705Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-11-03T16:15:21.9410025Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-11-03T16:15:21.9410347Z * [new branch] gh/PaulZhang12/32/base -> origin/gh/PaulZhang12/32/base 2025-11-03T16:15:21.9410701Z * [new branch] gh/PaulZhang12/32/head -> origin/gh/PaulZhang12/32/head 2025-11-03T16:15:21.9411021Z * [new branch] gh/PaulZhang12/32/orig -> origin/gh/PaulZhang12/32/orig 2025-11-03T16:15:21.9413510Z * [new branch] gh/PaulZhang12/33/base -> origin/gh/PaulZhang12/33/base 2025-11-03T16:15:21.9413822Z * [new branch] gh/PaulZhang12/33/head -> origin/gh/PaulZhang12/33/head 2025-11-03T16:15:21.9414133Z * [new branch] gh/PaulZhang12/33/orig -> origin/gh/PaulZhang12/33/orig 2025-11-03T16:15:21.9419252Z * [new branch] gh/PaulZhang12/34/base -> origin/gh/PaulZhang12/34/base 2025-11-03T16:15:21.9423769Z * [new branch] gh/PaulZhang12/34/head -> origin/gh/PaulZhang12/34/head 2025-11-03T16:15:21.9428692Z * [new branch] gh/PaulZhang12/34/orig -> origin/gh/PaulZhang12/34/orig 2025-11-03T16:15:21.9430450Z * [new branch] gh/PaulZhang12/35/base -> origin/gh/PaulZhang12/35/base 2025-11-03T16:15:21.9430843Z * [new branch] gh/PaulZhang12/35/head -> origin/gh/PaulZhang12/35/head 2025-11-03T16:15:21.9431199Z * [new branch] gh/PaulZhang12/35/orig -> origin/gh/PaulZhang12/35/orig 2025-11-03T16:15:21.9431538Z * [new branch] gh/PaulZhang12/36/base -> origin/gh/PaulZhang12/36/base 2025-11-03T16:15:21.9431889Z * [new branch] gh/PaulZhang12/36/head -> origin/gh/PaulZhang12/36/head 2025-11-03T16:15:21.9432256Z * [new branch] gh/PaulZhang12/36/orig -> origin/gh/PaulZhang12/36/orig 2025-11-03T16:15:21.9432601Z * [new branch] gh/PaulZhang12/37/base -> origin/gh/PaulZhang12/37/base 2025-11-03T16:15:21.9433152Z * [new branch] gh/PaulZhang12/37/head -> origin/gh/PaulZhang12/37/head 2025-11-03T16:15:21.9433491Z * [new branch] gh/PaulZhang12/37/orig -> origin/gh/PaulZhang12/37/orig 2025-11-03T16:15:21.9433832Z * [new branch] gh/PaulZhang12/38/base -> origin/gh/PaulZhang12/38/base 2025-11-03T16:15:21.9434309Z * [new branch] gh/PaulZhang12/38/head -> origin/gh/PaulZhang12/38/head 2025-11-03T16:15:21.9434657Z * [new branch] gh/PaulZhang12/38/orig -> origin/gh/PaulZhang12/38/orig 2025-11-03T16:15:21.9435016Z * [new branch] gh/PaulZhang12/39/base -> origin/gh/PaulZhang12/39/base 2025-11-03T16:15:21.9435370Z * [new branch] gh/PaulZhang12/39/head -> origin/gh/PaulZhang12/39/head 2025-11-03T16:15:21.9435735Z * [new branch] gh/PaulZhang12/39/orig -> origin/gh/PaulZhang12/39/orig 2025-11-03T16:15:21.9436107Z * [new branch] gh/PaulZhang12/40/base -> origin/gh/PaulZhang12/40/base 2025-11-03T16:15:21.9436455Z * [new branch] gh/PaulZhang12/40/head -> origin/gh/PaulZhang12/40/head 2025-11-03T16:15:21.9436795Z * [new branch] gh/PaulZhang12/40/orig -> origin/gh/PaulZhang12/40/orig 2025-11-03T16:15:21.9437179Z * [new branch] gh/PaulZhang12/41/base -> origin/gh/PaulZhang12/41/base 2025-11-03T16:15:21.9437536Z * [new branch] gh/PaulZhang12/41/head -> origin/gh/PaulZhang12/41/head 2025-11-03T16:15:21.9437948Z * [new branch] gh/PaulZhang12/41/orig -> origin/gh/PaulZhang12/41/orig 2025-11-03T16:15:21.9438295Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-11-03T16:15:21.9438767Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-11-03T16:15:21.9439135Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-11-03T16:15:21.9445515Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-11-03T16:15:21.9450000Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-11-03T16:15:21.9454483Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-11-03T16:15:21.9456636Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-11-03T16:15:21.9457154Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-11-03T16:15:21.9460633Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-11-03T16:15:21.9461039Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-11-03T16:15:21.9461433Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-11-03T16:15:21.9461782Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-11-03T16:15:21.9462144Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-11-03T16:15:21.9462496Z * [new branch] gh/SherlockNoMad/13/base -> origin/gh/SherlockNoMad/13/base 2025-11-03T16:15:21.9462833Z * [new branch] gh/SherlockNoMad/13/head -> origin/gh/SherlockNoMad/13/head 2025-11-03T16:15:21.9463163Z * [new branch] gh/SherlockNoMad/13/orig -> origin/gh/SherlockNoMad/13/orig 2025-11-03T16:15:21.9463499Z * [new branch] gh/SherlockNoMad/14/base -> origin/gh/SherlockNoMad/14/base 2025-11-03T16:15:21.9463834Z * [new branch] gh/SherlockNoMad/14/head -> origin/gh/SherlockNoMad/14/head 2025-11-03T16:15:21.9464168Z * [new branch] gh/SherlockNoMad/14/orig -> origin/gh/SherlockNoMad/14/orig 2025-11-03T16:15:21.9464494Z * [new branch] gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base 2025-11-03T16:15:21.9464965Z * [new branch] gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head 2025-11-03T16:15:21.9465307Z * [new branch] gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig 2025-11-03T16:15:21.9465653Z * [new branch] gh/SherlockNoMad/16/base -> origin/gh/SherlockNoMad/16/base 2025-11-03T16:15:21.9466008Z * [new branch] gh/SherlockNoMad/16/head -> origin/gh/SherlockNoMad/16/head 2025-11-03T16:15:21.9466336Z * [new branch] gh/SherlockNoMad/16/orig -> origin/gh/SherlockNoMad/16/orig 2025-11-03T16:15:21.9466671Z * [new branch] gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base 2025-11-03T16:15:21.9467007Z * [new branch] gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head 2025-11-03T16:15:21.9467349Z * [new branch] gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig 2025-11-03T16:15:21.9467705Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-11-03T16:15:21.9468054Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-11-03T16:15:21.9468407Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-11-03T16:15:21.9468741Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-11-03T16:15:21.9469075Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-11-03T16:15:21.9469413Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-11-03T16:15:21.9469750Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-11-03T16:15:21.9470089Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-11-03T16:15:21.9470456Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-11-03T16:15:21.9470894Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-11-03T16:15:21.9471273Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-11-03T16:15:21.9471680Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-11-03T16:15:21.9472114Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-11-03T16:15:21.9472481Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-11-03T16:15:21.9472837Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-11-03T16:15:21.9473203Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-11-03T16:15:21.9473550Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-11-03T16:15:21.9474020Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-11-03T16:15:21.9474448Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-11-03T16:15:21.9474819Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-11-03T16:15:21.9475201Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-11-03T16:15:21.9475559Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-11-03T16:15:21.9475908Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-11-03T16:15:21.9476261Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-11-03T16:15:21.9476614Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-11-03T16:15:21.9476974Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-11-03T16:15:21.9477389Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-11-03T16:15:21.9477753Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-11-03T16:15:21.9478099Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-11-03T16:15:21.9478449Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-11-03T16:15:21.9478798Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-11-03T16:15:21.9479139Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-11-03T16:15:21.9479493Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-11-03T16:15:21.9479812Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-11-03T16:15:21.9480143Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-11-03T16:15:21.9480490Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-11-03T16:15:21.9480821Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-11-03T16:15:21.9481170Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-11-03T16:15:21.9481515Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-11-03T16:15:21.9482178Z * [new branch] gh/XilunWu/166/base -> origin/gh/XilunWu/166/base 2025-11-03T16:15:21.9482741Z * [new branch] gh/XilunWu/166/head -> origin/gh/XilunWu/166/head 2025-11-03T16:15:21.9483397Z * [new branch] gh/XilunWu/166/orig -> origin/gh/XilunWu/166/orig 2025-11-03T16:15:21.9485584Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-11-03T16:15:21.9485992Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-11-03T16:15:21.9486388Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-11-03T16:15:21.9486949Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-11-03T16:15:21.9487575Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-11-03T16:15:21.9488294Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-11-03T16:15:21.9489365Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-11-03T16:15:21.9489805Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-11-03T16:15:21.9490347Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-11-03T16:15:21.9491551Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-11-03T16:15:21.9491941Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-11-03T16:15:21.9492520Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-11-03T16:15:21.9493711Z * [new branch] gh/XilunWu/172/base -> origin/gh/XilunWu/172/base 2025-11-03T16:15:21.9494215Z * [new branch] gh/XilunWu/172/head -> origin/gh/XilunWu/172/head 2025-11-03T16:15:21.9496710Z * [new branch] gh/XilunWu/172/orig -> origin/gh/XilunWu/172/orig 2025-11-03T16:15:21.9497082Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-11-03T16:15:21.9497397Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-11-03T16:15:21.9497696Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-11-03T16:15:21.9498006Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-11-03T16:15:21.9498346Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-11-03T16:15:21.9499362Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-11-03T16:15:21.9499963Z * [new branch] gh/XilunWu/176/base -> origin/gh/XilunWu/176/base 2025-11-03T16:15:21.9500568Z * [new branch] gh/XilunWu/176/head -> origin/gh/XilunWu/176/head 2025-11-03T16:15:21.9501186Z * [new branch] gh/XilunWu/176/orig -> origin/gh/XilunWu/176/orig 2025-11-03T16:15:21.9502297Z * [new branch] gh/XilunWu/177/base -> origin/gh/XilunWu/177/base 2025-11-03T16:15:21.9502604Z * [new branch] gh/XilunWu/177/head -> origin/gh/XilunWu/177/head 2025-11-03T16:15:21.9503513Z * [new branch] gh/XilunWu/177/orig -> origin/gh/XilunWu/177/orig 2025-11-03T16:15:21.9504960Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-11-03T16:15:21.9505291Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-11-03T16:15:21.9505631Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-11-03T16:15:21.9506690Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-11-03T16:15:21.9507122Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-11-03T16:15:21.9507808Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-11-03T16:15:21.9508920Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-11-03T16:15:21.9509349Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-11-03T16:15:21.9509919Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-11-03T16:15:21.9510916Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-11-03T16:15:21.9511283Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-11-03T16:15:21.9512260Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-11-03T16:15:21.9512937Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-11-03T16:15:21.9513573Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-11-03T16:15:21.9518019Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-11-03T16:15:21.9518404Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-11-03T16:15:21.9518734Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-11-03T16:15:21.9522475Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-11-03T16:15:21.9528211Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-11-03T16:15:21.9533112Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-11-03T16:15:21.9537509Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-11-03T16:15:21.9542515Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-11-03T16:15:21.9546921Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-11-03T16:15:21.9550369Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-11-03T16:15:21.9550767Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-11-03T16:15:21.9551080Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-11-03T16:15:21.9551395Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-11-03T16:15:21.9551705Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-11-03T16:15:21.9552213Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-11-03T16:15:21.9552544Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-11-03T16:15:21.9552863Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-11-03T16:15:21.9553182Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-11-03T16:15:21.9553501Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-11-03T16:15:21.9553827Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-11-03T16:15:21.9554274Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-11-03T16:15:21.9554589Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-11-03T16:15:21.9554939Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-11-03T16:15:21.9555266Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-11-03T16:15:21.9555601Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-11-03T16:15:21.9555922Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-11-03T16:15:21.9556230Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-11-03T16:15:21.9556546Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-11-03T16:15:21.9556859Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-11-03T16:15:21.9557171Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-11-03T16:15:21.9557487Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-11-03T16:15:21.9557798Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-11-03T16:15:21.9558116Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-11-03T16:15:21.9558498Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-11-03T16:15:21.9558812Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-11-03T16:15:21.9559122Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-11-03T16:15:21.9559439Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-11-03T16:15:21.9559752Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-11-03T16:15:21.9560067Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-11-03T16:15:21.9560380Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-11-03T16:15:21.9560687Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-11-03T16:15:21.9561013Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-11-03T16:15:21.9561328Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-11-03T16:15:21.9561642Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-11-03T16:15:21.9561954Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-11-03T16:15:21.9562265Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-11-03T16:15:21.9562573Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-11-03T16:15:21.9562882Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-11-03T16:15:21.9563228Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-11-03T16:15:21.9563542Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-11-03T16:15:21.9563846Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-11-03T16:15:21.9564167Z * [new branch] gh/XuehaiPan/384/base -> origin/gh/XuehaiPan/384/base 2025-11-03T16:15:21.9564476Z * [new branch] gh/XuehaiPan/384/head -> origin/gh/XuehaiPan/384/head 2025-11-03T16:15:21.9564791Z * [new branch] gh/XuehaiPan/384/orig -> origin/gh/XuehaiPan/384/orig 2025-11-03T16:15:21.9565102Z * [new branch] gh/XuehaiPan/390/base -> origin/gh/XuehaiPan/390/base 2025-11-03T16:15:21.9565409Z * [new branch] gh/XuehaiPan/390/head -> origin/gh/XuehaiPan/390/head 2025-11-03T16:15:21.9565723Z * [new branch] gh/XuehaiPan/390/orig -> origin/gh/XuehaiPan/390/orig 2025-11-03T16:15:21.9566037Z * [new branch] gh/XuehaiPan/391/base -> origin/gh/XuehaiPan/391/base 2025-11-03T16:15:21.9566354Z * [new branch] gh/XuehaiPan/391/head -> origin/gh/XuehaiPan/391/head 2025-11-03T16:15:21.9566673Z * [new branch] gh/XuehaiPan/391/orig -> origin/gh/XuehaiPan/391/orig 2025-11-03T16:15:21.9566985Z * [new branch] gh/XuehaiPan/392/base -> origin/gh/XuehaiPan/392/base 2025-11-03T16:15:21.9567294Z * [new branch] gh/XuehaiPan/392/head -> origin/gh/XuehaiPan/392/head 2025-11-03T16:15:21.9567603Z * [new branch] gh/XuehaiPan/392/orig -> origin/gh/XuehaiPan/392/orig 2025-11-03T16:15:21.9567915Z * [new branch] gh/XuehaiPan/393/base -> origin/gh/XuehaiPan/393/base 2025-11-03T16:15:21.9568227Z * [new branch] gh/XuehaiPan/393/head -> origin/gh/XuehaiPan/393/head 2025-11-03T16:15:21.9568536Z * [new branch] gh/XuehaiPan/393/orig -> origin/gh/XuehaiPan/393/orig 2025-11-03T16:15:21.9568848Z * [new branch] gh/XuehaiPan/394/base -> origin/gh/XuehaiPan/394/base 2025-11-03T16:15:21.9569164Z * [new branch] gh/XuehaiPan/394/head -> origin/gh/XuehaiPan/394/head 2025-11-03T16:15:21.9569529Z * [new branch] gh/XuehaiPan/394/orig -> origin/gh/XuehaiPan/394/orig 2025-11-03T16:15:21.9569839Z * [new branch] gh/XuehaiPan/395/base -> origin/gh/XuehaiPan/395/base 2025-11-03T16:15:21.9570144Z * [new branch] gh/XuehaiPan/395/head -> origin/gh/XuehaiPan/395/head 2025-11-03T16:15:21.9574478Z * [new branch] gh/XuehaiPan/395/orig -> origin/gh/XuehaiPan/395/orig 2025-11-03T16:15:21.9577377Z * [new branch] gh/XuehaiPan/396/base -> origin/gh/XuehaiPan/396/base 2025-11-03T16:15:21.9580128Z * [new branch] gh/XuehaiPan/396/orig -> origin/gh/XuehaiPan/396/orig 2025-11-03T16:15:21.9580634Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-11-03T16:15:21.9585817Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-11-03T16:15:21.9587965Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-11-03T16:15:21.9588538Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-11-03T16:15:21.9588998Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-11-03T16:15:21.9589447Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-11-03T16:15:21.9589775Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-11-03T16:15:21.9590121Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-11-03T16:15:21.9590436Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-11-03T16:15:21.9590914Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-11-03T16:15:21.9591256Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-11-03T16:15:21.9591594Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-11-03T16:15:21.9591929Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-11-03T16:15:21.9592261Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-11-03T16:15:21.9592595Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-11-03T16:15:21.9592922Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-11-03T16:15:21.9593250Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-11-03T16:15:21.9593573Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-11-03T16:15:21.9594013Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-11-03T16:15:21.9594361Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-11-03T16:15:21.9594693Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-11-03T16:15:21.9595026Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-11-03T16:15:21.9595415Z * [new branch] gh/aakhundov/3/base -> origin/gh/aakhundov/3/base 2025-11-03T16:15:21.9595738Z * [new branch] gh/aakhundov/3/head -> origin/gh/aakhundov/3/head 2025-11-03T16:15:21.9596059Z * [new branch] gh/aakhundov/3/orig -> origin/gh/aakhundov/3/orig 2025-11-03T16:15:21.9596396Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-11-03T16:15:21.9596731Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-11-03T16:15:21.9597041Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-11-03T16:15:21.9597358Z * [new branch] gh/albanD/1/base -> origin/gh/albanD/1/base 2025-11-03T16:15:21.9597731Z * [new branch] gh/albanD/1/head -> origin/gh/albanD/1/head 2025-11-03T16:15:21.9598031Z * [new branch] gh/albanD/1/orig -> origin/gh/albanD/1/orig 2025-11-03T16:15:21.9598330Z * [new branch] gh/albanD/2/base -> origin/gh/albanD/2/base 2025-11-03T16:15:21.9598618Z * [new branch] gh/albanD/2/head -> origin/gh/albanD/2/head 2025-11-03T16:15:21.9598917Z * [new branch] gh/albanD/2/orig -> origin/gh/albanD/2/orig 2025-11-03T16:15:21.9599216Z * [new branch] gh/albanD/3/base -> origin/gh/albanD/3/base 2025-11-03T16:15:21.9599521Z * [new branch] gh/albanD/3/head -> origin/gh/albanD/3/head 2025-11-03T16:15:21.9599820Z * [new branch] gh/albanD/3/orig -> origin/gh/albanD/3/orig 2025-11-03T16:15:21.9600109Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-11-03T16:15:21.9600410Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-11-03T16:15:21.9600711Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-11-03T16:15:21.9601165Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-11-03T16:15:21.9601624Z * [new branch] gh/alexsamardzic/11/base -> origin/gh/alexsamardzic/11/base 2025-11-03T16:15:21.9601970Z * [new branch] gh/alexsamardzic/11/head -> origin/gh/alexsamardzic/11/head 2025-11-03T16:15:21.9602476Z * [new branch] gh/alexsamardzic/11/orig -> origin/gh/alexsamardzic/11/orig 2025-11-03T16:15:21.9603016Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-11-03T16:15:21.9603488Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-11-03T16:15:21.9603966Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-11-03T16:15:21.9604363Z * [new branch] gh/alexsamardzic/13/base -> origin/gh/alexsamardzic/13/base 2025-11-03T16:15:21.9604683Z * [new branch] gh/alexsamardzic/13/head -> origin/gh/alexsamardzic/13/head 2025-11-03T16:15:21.9605372Z * [new branch] gh/alexsamardzic/13/orig -> origin/gh/alexsamardzic/13/orig 2025-11-03T16:15:21.9606061Z * [new branch] gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base 2025-11-03T16:15:21.9606612Z * [new branch] gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head 2025-11-03T16:15:21.9607233Z * [new branch] gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig 2025-11-03T16:15:21.9609005Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-11-03T16:15:21.9609528Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-11-03T16:15:21.9609972Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-11-03T16:15:21.9610605Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-11-03T16:15:21.9614243Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-11-03T16:15:21.9614785Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-11-03T16:15:21.9615266Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-11-03T16:15:21.9616140Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-11-03T16:15:21.9616675Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-11-03T16:15:21.9617162Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-11-03T16:15:21.9617689Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-11-03T16:15:21.9618016Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-11-03T16:15:21.9618549Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-11-03T16:15:21.9622456Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-11-03T16:15:21.9622982Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-11-03T16:15:21.9623449Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-11-03T16:15:21.9624198Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-11-03T16:15:21.9624581Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-11-03T16:15:21.9624906Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-11-03T16:15:21.9625389Z * [new branch] gh/andyanwang/40/base -> origin/gh/andyanwang/40/base 2025-11-03T16:15:21.9625713Z * [new branch] gh/andyanwang/40/head -> origin/gh/andyanwang/40/head 2025-11-03T16:15:21.9626097Z * [new branch] gh/andyanwang/40/orig -> origin/gh/andyanwang/40/orig 2025-11-03T16:15:21.9627501Z * [new branch] gh/andyanwang/41/base -> origin/gh/andyanwang/41/base 2025-11-03T16:15:21.9627927Z * [new branch] gh/andyanwang/41/head -> origin/gh/andyanwang/41/head 2025-11-03T16:15:21.9628336Z * [new branch] gh/andyanwang/41/orig -> origin/gh/andyanwang/41/orig 2025-11-03T16:15:21.9630787Z * [new branch] gh/andyanwang/42/base -> origin/gh/andyanwang/42/base 2025-11-03T16:15:21.9631214Z * [new branch] gh/andyanwang/42/head -> origin/gh/andyanwang/42/head 2025-11-03T16:15:21.9631559Z * [new branch] gh/andyanwang/42/orig -> origin/gh/andyanwang/42/orig 2025-11-03T16:15:21.9631910Z * [new branch] gh/andyanwang/43/base -> origin/gh/andyanwang/43/base 2025-11-03T16:15:21.9632298Z * [new branch] gh/andyanwang/43/head -> origin/gh/andyanwang/43/head 2025-11-03T16:15:21.9633066Z * [new branch] gh/andyanwang/43/orig -> origin/gh/andyanwang/43/orig 2025-11-03T16:15:21.9634270Z * [new branch] gh/andyanwang/44/base -> origin/gh/andyanwang/44/base 2025-11-03T16:15:21.9634618Z * [new branch] gh/andyanwang/44/head -> origin/gh/andyanwang/44/head 2025-11-03T16:15:21.9635216Z * [new branch] gh/andyanwang/44/orig -> origin/gh/andyanwang/44/orig 2025-11-03T16:15:21.9637706Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-11-03T16:15:21.9638366Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-11-03T16:15:21.9638789Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-11-03T16:15:21.9639194Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-11-03T16:15:21.9639583Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-11-03T16:15:21.9640188Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-11-03T16:15:21.9640684Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-11-03T16:15:21.9641399Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-11-03T16:15:21.9642167Z * [new branch] gh/angelayi/119/base -> origin/gh/angelayi/119/base 2025-11-03T16:15:21.9642826Z * [new branch] gh/angelayi/119/head -> origin/gh/angelayi/119/head 2025-11-03T16:15:21.9643557Z * [new branch] gh/angelayi/119/orig -> origin/gh/angelayi/119/orig 2025-11-03T16:15:21.9644460Z * [new branch] gh/angelayi/120/base -> origin/gh/angelayi/120/base 2025-11-03T16:15:21.9644994Z * [new branch] gh/angelayi/120/head -> origin/gh/angelayi/120/head 2025-11-03T16:15:21.9645558Z * [new branch] gh/angelayi/120/orig -> origin/gh/angelayi/120/orig 2025-11-03T16:15:21.9650001Z * [new branch] gh/angelayi/121/base -> origin/gh/angelayi/121/base 2025-11-03T16:15:21.9650552Z * [new branch] gh/angelayi/121/head -> origin/gh/angelayi/121/head 2025-11-03T16:15:21.9651416Z * [new branch] gh/angelayi/121/orig -> origin/gh/angelayi/121/orig 2025-11-03T16:15:21.9651957Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-11-03T16:15:21.9652411Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-11-03T16:15:21.9652725Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-11-03T16:15:21.9653037Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-11-03T16:15:21.9653335Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-11-03T16:15:21.9653670Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-11-03T16:15:21.9653983Z * [new branch] gh/angelayi/127/base -> origin/gh/angelayi/127/base 2025-11-03T16:15:21.9654295Z * [new branch] gh/angelayi/127/head -> origin/gh/angelayi/127/head 2025-11-03T16:15:21.9654641Z * [new branch] gh/angelayi/127/orig -> origin/gh/angelayi/127/orig 2025-11-03T16:15:21.9657271Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-11-03T16:15:21.9657816Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-11-03T16:15:21.9658227Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-11-03T16:15:21.9658949Z * [new branch] gh/angelayi/129/base -> origin/gh/angelayi/129/base 2025-11-03T16:15:21.9659319Z * [new branch] gh/angelayi/129/head -> origin/gh/angelayi/129/head 2025-11-03T16:15:21.9659642Z * [new branch] gh/angelayi/129/orig -> origin/gh/angelayi/129/orig 2025-11-03T16:15:21.9659992Z * [new branch] gh/angelayi/130/base -> origin/gh/angelayi/130/base 2025-11-03T16:15:21.9660632Z * [new branch] gh/angelayi/130/head -> origin/gh/angelayi/130/head 2025-11-03T16:15:21.9661240Z * [new branch] gh/angelayi/130/orig -> origin/gh/angelayi/130/orig 2025-11-03T16:15:21.9665182Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-11-03T16:15:21.9665577Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-11-03T16:15:21.9665923Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-11-03T16:15:21.9666255Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-11-03T16:15:21.9666583Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-11-03T16:15:21.9666913Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-11-03T16:15:21.9667230Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-11-03T16:15:21.9667558Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-11-03T16:15:21.9667923Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-11-03T16:15:21.9669139Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-11-03T16:15:21.9669477Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-11-03T16:15:21.9670142Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-11-03T16:15:21.9671110Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-11-03T16:15:21.9671659Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-11-03T16:15:21.9672335Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-11-03T16:15:21.9673536Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-11-03T16:15:21.9673878Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-11-03T16:15:21.9674623Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-11-03T16:15:21.9678469Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-11-03T16:15:21.9678997Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-11-03T16:15:21.9684257Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-11-03T16:15:21.9686894Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-11-03T16:15:21.9692269Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-11-03T16:15:21.9696747Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-11-03T16:15:21.9700334Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-11-03T16:15:21.9704622Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-11-03T16:15:21.9705043Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-11-03T16:15:21.9705357Z * [new branch] gh/anijain2305/891/base -> origin/gh/anijain2305/891/base 2025-11-03T16:15:21.9705686Z * [new branch] gh/anijain2305/891/head -> origin/gh/anijain2305/891/head 2025-11-03T16:15:21.9706012Z * [new branch] gh/anijain2305/891/orig -> origin/gh/anijain2305/891/orig 2025-11-03T16:15:21.9706332Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-11-03T16:15:21.9706662Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-11-03T16:15:21.9706970Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-11-03T16:15:21.9707287Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-11-03T16:15:21.9707614Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-11-03T16:15:21.9707950Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-11-03T16:15:21.9708278Z * [new branch] gh/anijain2305/896/base -> origin/gh/anijain2305/896/base 2025-11-03T16:15:21.9708594Z * [new branch] gh/anijain2305/896/head -> origin/gh/anijain2305/896/head 2025-11-03T16:15:21.9708918Z * [new branch] gh/anijain2305/896/orig -> origin/gh/anijain2305/896/orig 2025-11-03T16:15:21.9709240Z * [new branch] gh/anijain2305/897/base -> origin/gh/anijain2305/897/base 2025-11-03T16:15:21.9709562Z * [new branch] gh/anijain2305/897/head -> origin/gh/anijain2305/897/head 2025-11-03T16:15:21.9709882Z * [new branch] gh/anijain2305/897/orig -> origin/gh/anijain2305/897/orig 2025-11-03T16:15:21.9710191Z * [new branch] gh/anijain2305/898/base -> origin/gh/anijain2305/898/base 2025-11-03T16:15:21.9710515Z * [new branch] gh/anijain2305/898/head -> origin/gh/anijain2305/898/head 2025-11-03T16:15:21.9710835Z * [new branch] gh/anijain2305/898/orig -> origin/gh/anijain2305/898/orig 2025-11-03T16:15:21.9711200Z * [new branch] gh/anijain2305/899/base -> origin/gh/anijain2305/899/base 2025-11-03T16:15:21.9711521Z * [new branch] gh/anijain2305/899/head -> origin/gh/anijain2305/899/head 2025-11-03T16:15:21.9711832Z * [new branch] gh/anijain2305/899/orig -> origin/gh/anijain2305/899/orig 2025-11-03T16:15:21.9712152Z * [new branch] gh/anijain2305/900/base -> origin/gh/anijain2305/900/base 2025-11-03T16:15:21.9712474Z * [new branch] gh/anijain2305/900/head -> origin/gh/anijain2305/900/head 2025-11-03T16:15:21.9712806Z * [new branch] gh/anijain2305/900/orig -> origin/gh/anijain2305/900/orig 2025-11-03T16:15:21.9713146Z * [new branch] gh/anijain2305/901/base -> origin/gh/anijain2305/901/base 2025-11-03T16:15:21.9713648Z * [new branch] gh/anijain2305/901/head -> origin/gh/anijain2305/901/head 2025-11-03T16:15:21.9714135Z * [new branch] gh/anijain2305/901/orig -> origin/gh/anijain2305/901/orig 2025-11-03T16:15:21.9714507Z * [new branch] gh/anijain2305/902/base -> origin/gh/anijain2305/902/base 2025-11-03T16:15:21.9714868Z * [new branch] gh/anijain2305/902/head -> origin/gh/anijain2305/902/head 2025-11-03T16:15:21.9715227Z * [new branch] gh/anijain2305/902/orig -> origin/gh/anijain2305/902/orig 2025-11-03T16:15:21.9715560Z * [new branch] gh/anijain2305/903/base -> origin/gh/anijain2305/903/base 2025-11-03T16:15:21.9715882Z * [new branch] gh/anijain2305/903/head -> origin/gh/anijain2305/903/head 2025-11-03T16:15:21.9716207Z * [new branch] gh/anijain2305/903/orig -> origin/gh/anijain2305/903/orig 2025-11-03T16:15:21.9716626Z * [new branch] gh/anijain2305/904/base -> origin/gh/anijain2305/904/base 2025-11-03T16:15:21.9716953Z * [new branch] gh/anijain2305/904/head -> origin/gh/anijain2305/904/head 2025-11-03T16:15:21.9717273Z * [new branch] gh/anijain2305/904/orig -> origin/gh/anijain2305/904/orig 2025-11-03T16:15:21.9717597Z * [new branch] gh/anijain2305/905/base -> origin/gh/anijain2305/905/base 2025-11-03T16:15:21.9717918Z * [new branch] gh/anijain2305/905/head -> origin/gh/anijain2305/905/head 2025-11-03T16:15:21.9718238Z * [new branch] gh/anijain2305/905/orig -> origin/gh/anijain2305/905/orig 2025-11-03T16:15:21.9718560Z * [new branch] gh/anijain2305/906/base -> origin/gh/anijain2305/906/base 2025-11-03T16:15:21.9718873Z * [new branch] gh/anijain2305/906/head -> origin/gh/anijain2305/906/head 2025-11-03T16:15:21.9719195Z * [new branch] gh/anijain2305/906/orig -> origin/gh/anijain2305/906/orig 2025-11-03T16:15:21.9719516Z * [new branch] gh/anijain2305/907/base -> origin/gh/anijain2305/907/base 2025-11-03T16:15:21.9719844Z * [new branch] gh/anijain2305/907/head -> origin/gh/anijain2305/907/head 2025-11-03T16:15:21.9720164Z * [new branch] gh/anijain2305/907/orig -> origin/gh/anijain2305/907/orig 2025-11-03T16:15:21.9720507Z * [new branch] gh/anijain2305/908/base -> origin/gh/anijain2305/908/base 2025-11-03T16:15:21.9720827Z * [new branch] gh/anijain2305/908/head -> origin/gh/anijain2305/908/head 2025-11-03T16:15:21.9721186Z * [new branch] gh/anijain2305/908/orig -> origin/gh/anijain2305/908/orig 2025-11-03T16:15:21.9721512Z * [new branch] gh/anijain2305/909/base -> origin/gh/anijain2305/909/base 2025-11-03T16:15:21.9721847Z * [new branch] gh/anijain2305/909/head -> origin/gh/anijain2305/909/head 2025-11-03T16:15:21.9722159Z * [new branch] gh/anijain2305/909/orig -> origin/gh/anijain2305/909/orig 2025-11-03T16:15:21.9722680Z * [new branch] gh/anijain2305/910/base -> origin/gh/anijain2305/910/base 2025-11-03T16:15:21.9723185Z * [new branch] gh/anijain2305/910/head -> origin/gh/anijain2305/910/head 2025-11-03T16:15:21.9723653Z * [new branch] gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig 2025-11-03T16:15:21.9724334Z * [new branch] gh/anijain2305/911/base -> origin/gh/anijain2305/911/base 2025-11-03T16:15:21.9724932Z * [new branch] gh/anijain2305/911/head -> origin/gh/anijain2305/911/head 2025-11-03T16:15:21.9725572Z * [new branch] gh/anijain2305/911/orig -> origin/gh/anijain2305/911/orig 2025-11-03T16:15:21.9728383Z * [new branch] gh/anijain2305/912/base -> origin/gh/anijain2305/912/base 2025-11-03T16:15:21.9729044Z * [new branch] gh/anijain2305/912/head -> origin/gh/anijain2305/912/head 2025-11-03T16:15:21.9729574Z * [new branch] gh/anijain2305/912/orig -> origin/gh/anijain2305/912/orig 2025-11-03T16:15:21.9729956Z * [new branch] gh/anijain2305/913/base -> origin/gh/anijain2305/913/base 2025-11-03T16:15:21.9730376Z * [new branch] gh/anijain2305/913/head -> origin/gh/anijain2305/913/head 2025-11-03T16:15:21.9730766Z * [new branch] gh/anijain2305/913/orig -> origin/gh/anijain2305/913/orig 2025-11-03T16:15:21.9731371Z * [new branch] gh/anijain2305/914/base -> origin/gh/anijain2305/914/base 2025-11-03T16:15:21.9731939Z * [new branch] gh/anijain2305/914/head -> origin/gh/anijain2305/914/head 2025-11-03T16:15:21.9732503Z * [new branch] gh/anijain2305/914/orig -> origin/gh/anijain2305/914/orig 2025-11-03T16:15:21.9734088Z * [new branch] gh/anijain2305/915/base -> origin/gh/anijain2305/915/base 2025-11-03T16:15:21.9734570Z * [new branch] gh/anijain2305/915/head -> origin/gh/anijain2305/915/head 2025-11-03T16:15:21.9735026Z * [new branch] gh/anijain2305/915/orig -> origin/gh/anijain2305/915/orig 2025-11-03T16:15:21.9735507Z * [new branch] gh/anijain2305/916/base -> origin/gh/anijain2305/916/base 2025-11-03T16:15:21.9736219Z * [new branch] gh/anijain2305/916/head -> origin/gh/anijain2305/916/head 2025-11-03T16:15:21.9736831Z * [new branch] gh/anijain2305/916/orig -> origin/gh/anijain2305/916/orig 2025-11-03T16:15:21.9739364Z * [new branch] gh/anijain2305/917/base -> origin/gh/anijain2305/917/base 2025-11-03T16:15:21.9739916Z * [new branch] gh/anijain2305/917/head -> origin/gh/anijain2305/917/head 2025-11-03T16:15:21.9740370Z * [new branch] gh/anijain2305/917/orig -> origin/gh/anijain2305/917/orig 2025-11-03T16:15:21.9740801Z * [new branch] gh/anijain2305/918/base -> origin/gh/anijain2305/918/base 2025-11-03T16:15:21.9741531Z * [new branch] gh/anijain2305/918/head -> origin/gh/anijain2305/918/head 2025-11-03T16:15:21.9741929Z * [new branch] gh/anijain2305/918/orig -> origin/gh/anijain2305/918/orig 2025-11-03T16:15:21.9742304Z * [new branch] gh/anijain2305/919/base -> origin/gh/anijain2305/919/base 2025-11-03T16:15:21.9742818Z * [new branch] gh/anijain2305/919/head -> origin/gh/anijain2305/919/head 2025-11-03T16:15:21.9743440Z * [new branch] gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig 2025-11-03T16:15:21.9745742Z * [new branch] gh/anijain2305/920/base -> origin/gh/anijain2305/920/base 2025-11-03T16:15:21.9746292Z * [new branch] gh/anijain2305/920/head -> origin/gh/anijain2305/920/head 2025-11-03T16:15:21.9746733Z * [new branch] gh/anijain2305/920/orig -> origin/gh/anijain2305/920/orig 2025-11-03T16:15:21.9747498Z * [new branch] gh/anijain2305/921/base -> origin/gh/anijain2305/921/base 2025-11-03T16:15:21.9747883Z * [new branch] gh/anijain2305/921/head -> origin/gh/anijain2305/921/head 2025-11-03T16:15:21.9748351Z * [new branch] gh/anijain2305/921/orig -> origin/gh/anijain2305/921/orig 2025-11-03T16:15:21.9749011Z * [new branch] gh/anijain2305/922/base -> origin/gh/anijain2305/922/base 2025-11-03T16:15:21.9749588Z * [new branch] gh/anijain2305/922/head -> origin/gh/anijain2305/922/head 2025-11-03T16:15:21.9750250Z * [new branch] gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig 2025-11-03T16:15:21.9751176Z * [new branch] gh/anijain2305/923/base -> origin/gh/anijain2305/923/base 2025-11-03T16:15:21.9751714Z * [new branch] gh/anijain2305/923/head -> origin/gh/anijain2305/923/head 2025-11-03T16:15:21.9752367Z * [new branch] gh/anijain2305/923/orig -> origin/gh/anijain2305/923/orig 2025-11-03T16:15:21.9753133Z * [new branch] gh/anijain2305/924/base -> origin/gh/anijain2305/924/base 2025-11-03T16:15:21.9753784Z * [new branch] gh/anijain2305/924/head -> origin/gh/anijain2305/924/head 2025-11-03T16:15:21.9754472Z * [new branch] gh/anijain2305/924/orig -> origin/gh/anijain2305/924/orig 2025-11-03T16:15:21.9755781Z * [new branch] gh/anijain2305/925/base -> origin/gh/anijain2305/925/base 2025-11-03T16:15:21.9756125Z * [new branch] gh/anijain2305/925/head -> origin/gh/anijain2305/925/head 2025-11-03T16:15:21.9756721Z * [new branch] gh/anijain2305/925/orig -> origin/gh/anijain2305/925/orig 2025-11-03T16:15:21.9758539Z * [new branch] gh/anijain2305/926/base -> origin/gh/anijain2305/926/base 2025-11-03T16:15:21.9758941Z * [new branch] gh/anijain2305/926/head -> origin/gh/anijain2305/926/head 2025-11-03T16:15:21.9761763Z * [new branch] gh/anijain2305/926/orig -> origin/gh/anijain2305/926/orig 2025-11-03T16:15:21.9762128Z * [new branch] gh/anijain2305/927/base -> origin/gh/anijain2305/927/base 2025-11-03T16:15:21.9762483Z * [new branch] gh/anijain2305/927/head -> origin/gh/anijain2305/927/head 2025-11-03T16:15:21.9762814Z * [new branch] gh/anijain2305/927/orig -> origin/gh/anijain2305/927/orig 2025-11-03T16:15:21.9765063Z * [new branch] gh/anijain2305/928/base -> origin/gh/anijain2305/928/base 2025-11-03T16:15:21.9765403Z * [new branch] gh/anijain2305/928/head -> origin/gh/anijain2305/928/head 2025-11-03T16:15:21.9765742Z * [new branch] gh/anijain2305/928/orig -> origin/gh/anijain2305/928/orig 2025-11-03T16:15:21.9766073Z * [new branch] gh/anijain2305/929/base -> origin/gh/anijain2305/929/base 2025-11-03T16:15:21.9766411Z * [new branch] gh/anijain2305/929/head -> origin/gh/anijain2305/929/head 2025-11-03T16:15:21.9768506Z * [new branch] gh/anijain2305/929/orig -> origin/gh/anijain2305/929/orig 2025-11-03T16:15:21.9768930Z * [new branch] gh/anijain2305/930/base -> origin/gh/anijain2305/930/base 2025-11-03T16:15:21.9769358Z * [new branch] gh/anijain2305/930/head -> origin/gh/anijain2305/930/head 2025-11-03T16:15:21.9769781Z * [new branch] gh/anijain2305/930/orig -> origin/gh/anijain2305/930/orig 2025-11-03T16:15:21.9770534Z * [new branch] gh/anijain2305/931/base -> origin/gh/anijain2305/931/base 2025-11-03T16:15:21.9770977Z * [new branch] gh/anijain2305/931/head -> origin/gh/anijain2305/931/head 2025-11-03T16:15:21.9771361Z * [new branch] gh/anijain2305/931/orig -> origin/gh/anijain2305/931/orig 2025-11-03T16:15:21.9771739Z * [new branch] gh/anijain2305/932/base -> origin/gh/anijain2305/932/base 2025-11-03T16:15:21.9776423Z * [new branch] gh/anijain2305/932/head -> origin/gh/anijain2305/932/head 2025-11-03T16:15:21.9780560Z * [new branch] gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig 2025-11-03T16:15:21.9785077Z * [new branch] gh/anijain2305/933/base -> origin/gh/anijain2305/933/base 2025-11-03T16:15:21.9789735Z * [new branch] gh/anijain2305/933/head -> origin/gh/anijain2305/933/head 2025-11-03T16:15:21.9789945Z * [new branch] gh/anijain2305/933/orig -> origin/gh/anijain2305/933/orig 2025-11-03T16:15:21.9790099Z * [new branch] gh/anijain2305/934/base -> origin/gh/anijain2305/934/base 2025-11-03T16:15:21.9790264Z * [new branch] gh/anijain2305/934/head -> origin/gh/anijain2305/934/head 2025-11-03T16:15:21.9790404Z * [new branch] gh/anijain2305/934/orig -> origin/gh/anijain2305/934/orig 2025-11-03T16:15:21.9790559Z * [new branch] gh/anijain2305/935/base -> origin/gh/anijain2305/935/base 2025-11-03T16:15:21.9790703Z * [new branch] gh/anijain2305/935/head -> origin/gh/anijain2305/935/head 2025-11-03T16:15:21.9790840Z * [new branch] gh/anijain2305/935/orig -> origin/gh/anijain2305/935/orig 2025-11-03T16:15:21.9790992Z * [new branch] gh/anijain2305/936/base -> origin/gh/anijain2305/936/base 2025-11-03T16:15:21.9791127Z * [new branch] gh/anijain2305/936/head -> origin/gh/anijain2305/936/head 2025-11-03T16:15:21.9791288Z * [new branch] gh/anijain2305/936/orig -> origin/gh/anijain2305/936/orig 2025-11-03T16:15:21.9791425Z * [new branch] gh/anijain2305/937/base -> origin/gh/anijain2305/937/base 2025-11-03T16:15:21.9791558Z * [new branch] gh/anijain2305/937/head -> origin/gh/anijain2305/937/head 2025-11-03T16:15:21.9791704Z * [new branch] gh/anijain2305/937/orig -> origin/gh/anijain2305/937/orig 2025-11-03T16:15:21.9791993Z * [new branch] gh/anijain2305/938/base -> origin/gh/anijain2305/938/base 2025-11-03T16:15:21.9792148Z * [new branch] gh/anijain2305/938/head -> origin/gh/anijain2305/938/head 2025-11-03T16:15:21.9792286Z * [new branch] gh/anijain2305/938/orig -> origin/gh/anijain2305/938/orig 2025-11-03T16:15:21.9792431Z * [new branch] gh/anijain2305/939/base -> origin/gh/anijain2305/939/base 2025-11-03T16:15:21.9792567Z * [new branch] gh/anijain2305/939/head -> origin/gh/anijain2305/939/head 2025-11-03T16:15:21.9792703Z * [new branch] gh/anijain2305/939/orig -> origin/gh/anijain2305/939/orig 2025-11-03T16:15:21.9792849Z * [new branch] gh/anijain2305/940/base -> origin/gh/anijain2305/940/base 2025-11-03T16:15:21.9792998Z * [new branch] gh/anijain2305/940/head -> origin/gh/anijain2305/940/head 2025-11-03T16:15:21.9793142Z * [new branch] gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig 2025-11-03T16:15:21.9793289Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-11-03T16:15:21.9793429Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-11-03T16:15:21.9793721Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-11-03T16:15:21.9797263Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-11-03T16:15:21.9797406Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-11-03T16:15:21.9797536Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-11-03T16:15:21.9797752Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-11-03T16:15:21.9802579Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-11-03T16:15:21.9804546Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-11-03T16:15:21.9804709Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-11-03T16:15:21.9804839Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-11-03T16:15:21.9805123Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-11-03T16:15:21.9805249Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-11-03T16:15:21.9805385Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-11-03T16:15:21.9805511Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-11-03T16:15:21.9805636Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-11-03T16:15:21.9805785Z * [new branch] gh/anshul-si/51/base -> origin/gh/anshul-si/51/base 2025-11-03T16:15:21.9805924Z * [new branch] gh/anshul-si/51/head -> origin/gh/anshul-si/51/head 2025-11-03T16:15:21.9810000Z * [new branch] gh/anshul-si/51/orig -> origin/gh/anshul-si/51/orig 2025-11-03T16:15:21.9812237Z * [new branch] gh/anshul-si/52/base -> origin/gh/anshul-si/52/base 2025-11-03T16:15:21.9816674Z * [new branch] gh/anshul-si/52/head -> origin/gh/anshul-si/52/head 2025-11-03T16:15:21.9820875Z * [new branch] gh/anshul-si/52/orig -> origin/gh/anshul-si/52/orig 2025-11-03T16:15:21.9821054Z * [new branch] gh/anshul-si/53/base -> origin/gh/anshul-si/53/base 2025-11-03T16:15:21.9821183Z * [new branch] gh/anshul-si/53/head -> origin/gh/anshul-si/53/head 2025-11-03T16:15:21.9821305Z * [new branch] gh/anshul-si/54/base -> origin/gh/anshul-si/54/base 2025-11-03T16:15:21.9821434Z * [new branch] gh/anshul-si/54/head -> origin/gh/anshul-si/54/head 2025-11-03T16:15:21.9821757Z * [new branch] gh/anshul-si/54/orig -> origin/gh/anshul-si/54/orig 2025-11-03T16:15:21.9821892Z * [new branch] gh/anshul-si/55/base -> origin/gh/anshul-si/55/base 2025-11-03T16:15:21.9822033Z * [new branch] gh/anshul-si/55/head -> origin/gh/anshul-si/55/head 2025-11-03T16:15:21.9822160Z * [new branch] gh/anshul-si/55/orig -> origin/gh/anshul-si/55/orig 2025-11-03T16:15:21.9822280Z * [new branch] gh/anshul-si/56/base -> origin/gh/anshul-si/56/base 2025-11-03T16:15:21.9822398Z * [new branch] gh/anshul-si/56/head -> origin/gh/anshul-si/56/head 2025-11-03T16:15:21.9822525Z * [new branch] gh/anshul-si/56/orig -> origin/gh/anshul-si/56/orig 2025-11-03T16:15:21.9822644Z * [new branch] gh/anshul-si/57/base -> origin/gh/anshul-si/57/base 2025-11-03T16:15:21.9822770Z * [new branch] gh/anshul-si/57/head -> origin/gh/anshul-si/57/head 2025-11-03T16:15:21.9822893Z * [new branch] gh/anshul-si/57/orig -> origin/gh/anshul-si/57/orig 2025-11-03T16:15:21.9823012Z * [new branch] gh/anshul-si/58/base -> origin/gh/anshul-si/58/base 2025-11-03T16:15:21.9823141Z * [new branch] gh/anshul-si/58/head -> origin/gh/anshul-si/58/head 2025-11-03T16:15:21.9823259Z * [new branch] gh/anshul-si/59/base -> origin/gh/anshul-si/59/base 2025-11-03T16:15:21.9823384Z * [new branch] gh/anshul-si/59/head -> origin/gh/anshul-si/59/head 2025-11-03T16:15:21.9823502Z * [new branch] gh/anshul-si/59/orig -> origin/gh/anshul-si/59/orig 2025-11-03T16:15:21.9823632Z * [new branch] gh/anshul-si/60/base -> origin/gh/anshul-si/60/base 2025-11-03T16:15:21.9828152Z * [new branch] gh/anshul-si/60/head -> origin/gh/anshul-si/60/head 2025-11-03T16:15:21.9830431Z * [new branch] gh/anshul-si/60/orig -> origin/gh/anshul-si/60/orig 2025-11-03T16:15:21.9830753Z * [new branch] gh/anshul-si/61/base -> origin/gh/anshul-si/61/base 2025-11-03T16:15:21.9830934Z * [new branch] gh/anshul-si/61/head -> origin/gh/anshul-si/61/head 2025-11-03T16:15:21.9831360Z * [new branch] gh/anshul-si/61/orig -> origin/gh/anshul-si/61/orig 2025-11-03T16:15:21.9831574Z * [new branch] gh/anshul-si/62/base -> origin/gh/anshul-si/62/base 2025-11-03T16:15:21.9831725Z * [new branch] gh/anshul-si/62/head -> origin/gh/anshul-si/62/head 2025-11-03T16:15:21.9831854Z * [new branch] gh/anshul-si/62/orig -> origin/gh/anshul-si/62/orig 2025-11-03T16:15:21.9832110Z * [new branch] gh/anshul-si/63/base -> origin/gh/anshul-si/63/base 2025-11-03T16:15:21.9832732Z * [new branch] gh/anshul-si/63/head -> origin/gh/anshul-si/63/head 2025-11-03T16:15:21.9832930Z * [new branch] gh/anshul-si/63/orig -> origin/gh/anshul-si/63/orig 2025-11-03T16:15:21.9833064Z * [new branch] gh/anshul-si/64/base -> origin/gh/anshul-si/64/base 2025-11-03T16:15:21.9833225Z * [new branch] gh/anshul-si/64/head -> origin/gh/anshul-si/64/head 2025-11-03T16:15:21.9833351Z * [new branch] gh/anshul-si/64/orig -> origin/gh/anshul-si/64/orig 2025-11-03T16:15:21.9833482Z * [new branch] gh/anshul-si/65/base -> origin/gh/anshul-si/65/base 2025-11-03T16:15:21.9833607Z * [new branch] gh/anshul-si/65/head -> origin/gh/anshul-si/65/head 2025-11-03T16:15:21.9833743Z * [new branch] gh/anshul-si/65/orig -> origin/gh/anshul-si/65/orig 2025-11-03T16:15:21.9837865Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-11-03T16:15:21.9838185Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-11-03T16:15:21.9838530Z * [new branch] gh/aorenste/133/base -> origin/gh/aorenste/133/base 2025-11-03T16:15:21.9838791Z * [new branch] gh/aorenste/133/head -> origin/gh/aorenste/133/head 2025-11-03T16:15:21.9838964Z * [new branch] gh/aorenste/133/orig -> origin/gh/aorenste/133/orig 2025-11-03T16:15:21.9839636Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-11-03T16:15:21.9839810Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-11-03T16:15:21.9840069Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-11-03T16:15:21.9843831Z * [new branch] gh/aorenste/135/base -> origin/gh/aorenste/135/base 2025-11-03T16:15:21.9848928Z * [new branch] gh/aorenste/135/head -> origin/gh/aorenste/135/head 2025-11-03T16:15:21.9850657Z * [new branch] gh/aorenste/135/orig -> origin/gh/aorenste/135/orig 2025-11-03T16:15:21.9850947Z * [new branch] gh/aorenste/136/base -> origin/gh/aorenste/136/base 2025-11-03T16:15:21.9856504Z * [new branch] gh/aorenste/136/head -> origin/gh/aorenste/136/head 2025-11-03T16:15:21.9858585Z * [new branch] gh/aorenste/136/orig -> origin/gh/aorenste/136/orig 2025-11-03T16:15:21.9858851Z * [new branch] gh/aorenste/137/base -> origin/gh/aorenste/137/base 2025-11-03T16:15:21.9863237Z * [new branch] gh/aorenste/137/head -> origin/gh/aorenste/137/head 2025-11-03T16:15:21.9863493Z * [new branch] gh/aorenste/137/orig -> origin/gh/aorenste/137/orig 2025-11-03T16:15:21.9868944Z * [new branch] gh/aorenste/138/base -> origin/gh/aorenste/138/base 2025-11-03T16:15:21.9869131Z * [new branch] gh/aorenste/138/head -> origin/gh/aorenste/138/head 2025-11-03T16:15:21.9869276Z * [new branch] gh/aorenste/138/orig -> origin/gh/aorenste/138/orig 2025-11-03T16:15:21.9869404Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-11-03T16:15:21.9869542Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-11-03T16:15:21.9869813Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-11-03T16:15:21.9869950Z * [new branch] gh/aorenste/140/base -> origin/gh/aorenste/140/base 2025-11-03T16:15:21.9870081Z * [new branch] gh/aorenste/140/head -> origin/gh/aorenste/140/head 2025-11-03T16:15:21.9870219Z * [new branch] gh/aorenste/140/orig -> origin/gh/aorenste/140/orig 2025-11-03T16:15:21.9870347Z * [new branch] gh/aorenste/141/base -> origin/gh/aorenste/141/base 2025-11-03T16:15:21.9870477Z * [new branch] gh/aorenste/141/head -> origin/gh/aorenste/141/head 2025-11-03T16:15:21.9870620Z * [new branch] gh/aorenste/142/base -> origin/gh/aorenste/142/base 2025-11-03T16:15:21.9870750Z * [new branch] gh/aorenste/142/head -> origin/gh/aorenste/142/head 2025-11-03T16:15:21.9870892Z * [new branch] gh/aorenste/142/orig -> origin/gh/aorenste/142/orig 2025-11-03T16:15:21.9871018Z * [new branch] gh/aorenste/143/base -> origin/gh/aorenste/143/base 2025-11-03T16:15:21.9871154Z * [new branch] gh/aorenste/143/head -> origin/gh/aorenste/143/head 2025-11-03T16:15:21.9871282Z * [new branch] gh/aorenste/143/orig -> origin/gh/aorenste/143/orig 2025-11-03T16:15:21.9871407Z * [new branch] gh/aorenste/144/base -> origin/gh/aorenste/144/base 2025-11-03T16:15:21.9871537Z * [new branch] gh/aorenste/144/head -> origin/gh/aorenste/144/head 2025-11-03T16:15:21.9871662Z * [new branch] gh/aorenste/144/orig -> origin/gh/aorenste/144/orig 2025-11-03T16:15:21.9871849Z * [new branch] gh/aorenste/145/base -> origin/gh/aorenste/145/base 2025-11-03T16:15:21.9871979Z * [new branch] gh/aorenste/145/head -> origin/gh/aorenste/145/head 2025-11-03T16:15:21.9872109Z * [new branch] gh/aorenste/145/orig -> origin/gh/aorenste/145/orig 2025-11-03T16:15:21.9872273Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-11-03T16:15:21.9872416Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-11-03T16:15:21.9872565Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-11-03T16:15:21.9872703Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-11-03T16:15:21.9872849Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-11-03T16:15:21.9872987Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-11-03T16:15:21.9873119Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-11-03T16:15:21.9873262Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-11-03T16:15:21.9873392Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-11-03T16:15:21.9873524Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-11-03T16:15:21.9873654Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-11-03T16:15:21.9873797Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-11-03T16:15:21.9874953Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-11-03T16:15:21.9875361Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-11-03T16:15:21.9875529Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-11-03T16:15:21.9876042Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-11-03T16:15:21.9877255Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-11-03T16:15:21.9877910Z * [new branch] gh/bdhirsh/671/base -> origin/gh/bdhirsh/671/base 2025-11-03T16:15:21.9880273Z * [new branch] gh/bdhirsh/671/head -> origin/gh/bdhirsh/671/head 2025-11-03T16:15:21.9880603Z * [new branch] gh/bdhirsh/671/orig -> origin/gh/bdhirsh/671/orig 2025-11-03T16:15:21.9880791Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-11-03T16:15:21.9880934Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-11-03T16:15:21.9881503Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-11-03T16:15:21.9885492Z * [new branch] gh/bdhirsh/673/base -> origin/gh/bdhirsh/673/base 2025-11-03T16:15:21.9885825Z * [new branch] gh/bdhirsh/673/head -> origin/gh/bdhirsh/673/head 2025-11-03T16:15:21.9885999Z * [new branch] gh/bdhirsh/673/orig -> origin/gh/bdhirsh/673/orig 2025-11-03T16:15:21.9886182Z * [new branch] gh/bdhirsh/674/base -> origin/gh/bdhirsh/674/base 2025-11-03T16:15:21.9886394Z * [new branch] gh/bdhirsh/674/head -> origin/gh/bdhirsh/674/head 2025-11-03T16:15:21.9886543Z * [new branch] gh/bdhirsh/674/orig -> origin/gh/bdhirsh/674/orig 2025-11-03T16:15:21.9887552Z * [new branch] gh/bdhirsh/675/base -> origin/gh/bdhirsh/675/base 2025-11-03T16:15:21.9887861Z * [new branch] gh/bdhirsh/675/head -> origin/gh/bdhirsh/675/head 2025-11-03T16:15:21.9890416Z * [new branch] gh/bdhirsh/675/orig -> origin/gh/bdhirsh/675/orig 2025-11-03T16:15:21.9890917Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-11-03T16:15:21.9891163Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-11-03T16:15:21.9891345Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-11-03T16:15:21.9891905Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-11-03T16:15:21.9894995Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-11-03T16:15:21.9895326Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-11-03T16:15:21.9895578Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-11-03T16:15:21.9895800Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-11-03T16:15:21.9896046Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-11-03T16:15:21.9896738Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-11-03T16:15:21.9896928Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-11-03T16:15:21.9897372Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-11-03T16:15:21.9901000Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-11-03T16:15:21.9901191Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-11-03T16:15:21.9901335Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-11-03T16:15:21.9901481Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-11-03T16:15:21.9901632Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-11-03T16:15:21.9901791Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-11-03T16:15:21.9902200Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-11-03T16:15:21.9902775Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-11-03T16:15:21.9904218Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-11-03T16:15:21.9904469Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-11-03T16:15:21.9905364Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-11-03T16:15:21.9905928Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-11-03T16:15:21.9907067Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-11-03T16:15:21.9907341Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-11-03T16:15:21.9907821Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-11-03T16:15:21.9909504Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-11-03T16:15:21.9909680Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-11-03T16:15:21.9910118Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-11-03T16:15:21.9911528Z * [new branch] gh/bobrenjc93/625/base -> origin/gh/bobrenjc93/625/base 2025-11-03T16:15:21.9911677Z * [new branch] gh/bobrenjc93/625/head -> origin/gh/bobrenjc93/625/head 2025-11-03T16:15:21.9912336Z * [new branch] gh/bobrenjc93/625/orig -> origin/gh/bobrenjc93/625/orig 2025-11-03T16:15:21.9913761Z * [new branch] gh/bobrenjc93/626/base -> origin/gh/bobrenjc93/626/base 2025-11-03T16:15:21.9914218Z * [new branch] gh/bobrenjc93/626/head -> origin/gh/bobrenjc93/626/head 2025-11-03T16:15:21.9919899Z * [new branch] gh/bobrenjc93/626/orig -> origin/gh/bobrenjc93/626/orig 2025-11-03T16:15:21.9920180Z * [new branch] gh/bobrenjc93/627/base -> origin/gh/bobrenjc93/627/base 2025-11-03T16:15:21.9925720Z * [new branch] gh/bobrenjc93/627/head -> origin/gh/bobrenjc93/627/head 2025-11-03T16:15:21.9927305Z * [new branch] gh/bobrenjc93/627/orig -> origin/gh/bobrenjc93/627/orig 2025-11-03T16:15:21.9927590Z * [new branch] gh/bobrenjc93/630/base -> origin/gh/bobrenjc93/630/base 2025-11-03T16:15:21.9933389Z * [new branch] gh/bobrenjc93/630/head -> origin/gh/bobrenjc93/630/head 2025-11-03T16:15:21.9935089Z * [new branch] gh/bobrenjc93/630/orig -> origin/gh/bobrenjc93/630/orig 2025-11-03T16:15:21.9935430Z * [new branch] gh/bobrenjc93/631/base -> origin/gh/bobrenjc93/631/base 2025-11-03T16:15:21.9935659Z * [new branch] gh/bobrenjc93/631/head -> origin/gh/bobrenjc93/631/head 2025-11-03T16:15:21.9935814Z * [new branch] gh/bobrenjc93/631/orig -> origin/gh/bobrenjc93/631/orig 2025-11-03T16:15:21.9936048Z * [new branch] gh/bobrenjc93/632/base -> origin/gh/bobrenjc93/632/base 2025-11-03T16:15:21.9936195Z * [new branch] gh/bobrenjc93/632/head -> origin/gh/bobrenjc93/632/head 2025-11-03T16:15:21.9936412Z * [new branch] gh/bobrenjc93/632/orig -> origin/gh/bobrenjc93/632/orig 2025-11-03T16:15:21.9936563Z * [new branch] gh/bobrenjc93/633/base -> origin/gh/bobrenjc93/633/base 2025-11-03T16:15:21.9936705Z * [new branch] gh/bobrenjc93/633/head -> origin/gh/bobrenjc93/633/head 2025-11-03T16:15:21.9937415Z * [new branch] gh/bobrenjc93/633/orig -> origin/gh/bobrenjc93/633/orig 2025-11-03T16:15:21.9937613Z * [new branch] gh/bobrenjc93/634/base -> origin/gh/bobrenjc93/634/base 2025-11-03T16:15:21.9937761Z * [new branch] gh/bobrenjc93/634/head -> origin/gh/bobrenjc93/634/head 2025-11-03T16:15:21.9938089Z * [new branch] gh/bobrenjc93/634/orig -> origin/gh/bobrenjc93/634/orig 2025-11-03T16:15:21.9938225Z * [new branch] gh/bobrenjc93/635/base -> origin/gh/bobrenjc93/635/base 2025-11-03T16:15:21.9938380Z * [new branch] gh/bobrenjc93/635/head -> origin/gh/bobrenjc93/635/head 2025-11-03T16:15:21.9938513Z * [new branch] gh/bobrenjc93/635/orig -> origin/gh/bobrenjc93/635/orig 2025-11-03T16:15:21.9938652Z * [new branch] gh/bobrenjc93/636/base -> origin/gh/bobrenjc93/636/base 2025-11-03T16:15:21.9938784Z * [new branch] gh/bobrenjc93/636/head -> origin/gh/bobrenjc93/636/head 2025-11-03T16:15:21.9938925Z * [new branch] gh/bobrenjc93/636/orig -> origin/gh/bobrenjc93/636/orig 2025-11-03T16:15:21.9939056Z * [new branch] gh/bobrenjc93/637/base -> origin/gh/bobrenjc93/637/base 2025-11-03T16:15:21.9939189Z * [new branch] gh/bobrenjc93/637/head -> origin/gh/bobrenjc93/637/head 2025-11-03T16:15:21.9939329Z * [new branch] gh/bobrenjc93/637/orig -> origin/gh/bobrenjc93/637/orig 2025-11-03T16:15:21.9939460Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-11-03T16:15:21.9939603Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-11-03T16:15:21.9939732Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-11-03T16:15:21.9940757Z * [new branch] gh/bobrenjc93/639/base -> origin/gh/bobrenjc93/639/base 2025-11-03T16:15:21.9940985Z * [new branch] gh/bobrenjc93/639/head -> origin/gh/bobrenjc93/639/head 2025-11-03T16:15:21.9941479Z * [new branch] gh/bobrenjc93/639/orig -> origin/gh/bobrenjc93/639/orig 2025-11-03T16:15:21.9945500Z * [new branch] gh/bobrenjc93/640/base -> origin/gh/bobrenjc93/640/base 2025-11-03T16:15:21.9945838Z * [new branch] gh/bobrenjc93/640/head -> origin/gh/bobrenjc93/640/head 2025-11-03T16:15:21.9946063Z * [new branch] gh/bobrenjc93/640/orig -> origin/gh/bobrenjc93/640/orig 2025-11-03T16:15:21.9946226Z * [new branch] gh/bobrenjc93/641/base -> origin/gh/bobrenjc93/641/base 2025-11-03T16:15:21.9946358Z * [new branch] gh/bobrenjc93/641/head -> origin/gh/bobrenjc93/641/head 2025-11-03T16:15:21.9946612Z * [new branch] gh/bobrenjc93/641/orig -> origin/gh/bobrenjc93/641/orig 2025-11-03T16:15:21.9946759Z * [new branch] gh/bobrenjc93/642/base -> origin/gh/bobrenjc93/642/base 2025-11-03T16:15:21.9947616Z * [new branch] gh/bobrenjc93/642/head -> origin/gh/bobrenjc93/642/head 2025-11-03T16:15:21.9948268Z * [new branch] gh/bobrenjc93/642/orig -> origin/gh/bobrenjc93/642/orig 2025-11-03T16:15:21.9949174Z * [new branch] gh/bobrenjc93/643/base -> origin/gh/bobrenjc93/643/base 2025-11-03T16:15:21.9949479Z * [new branch] gh/bobrenjc93/643/head -> origin/gh/bobrenjc93/643/head 2025-11-03T16:15:21.9951479Z * [new branch] gh/bobrenjc93/643/orig -> origin/gh/bobrenjc93/643/orig 2025-11-03T16:15:21.9951650Z * [new branch] gh/bobrenjc93/644/base -> origin/gh/bobrenjc93/644/base 2025-11-03T16:15:21.9951788Z * [new branch] gh/bobrenjc93/644/head -> origin/gh/bobrenjc93/644/head 2025-11-03T16:15:21.9952252Z * [new branch] gh/bobrenjc93/644/orig -> origin/gh/bobrenjc93/644/orig 2025-11-03T16:15:21.9953463Z * [new branch] gh/bobrenjc93/645/base -> origin/gh/bobrenjc93/645/base 2025-11-03T16:15:21.9953722Z * [new branch] gh/bobrenjc93/645/head -> origin/gh/bobrenjc93/645/head 2025-11-03T16:15:21.9956371Z * [new branch] gh/bobrenjc93/645/orig -> origin/gh/bobrenjc93/645/orig 2025-11-03T16:15:21.9956688Z * [new branch] gh/bobrenjc93/646/base -> origin/gh/bobrenjc93/646/base 2025-11-03T16:15:21.9957104Z * [new branch] gh/bobrenjc93/646/head -> origin/gh/bobrenjc93/646/head 2025-11-03T16:15:21.9957255Z * [new branch] gh/bobrenjc93/646/orig -> origin/gh/bobrenjc93/646/orig 2025-11-03T16:15:21.9957640Z * [new branch] gh/bobrenjc93/647/base -> origin/gh/bobrenjc93/647/base 2025-11-03T16:15:21.9958427Z * [new branch] gh/bobrenjc93/647/head -> origin/gh/bobrenjc93/647/head 2025-11-03T16:15:21.9958818Z * [new branch] gh/bobrenjc93/647/orig -> origin/gh/bobrenjc93/647/orig 2025-11-03T16:15:21.9961761Z * [new branch] gh/bobrenjc93/648/base -> origin/gh/bobrenjc93/648/base 2025-11-03T16:15:21.9962088Z * [new branch] gh/bobrenjc93/648/head -> origin/gh/bobrenjc93/648/head 2025-11-03T16:15:21.9962323Z * [new branch] gh/bobrenjc93/648/orig -> origin/gh/bobrenjc93/648/orig 2025-11-03T16:15:21.9962479Z * [new branch] gh/bobrenjc93/649/base -> origin/gh/bobrenjc93/649/base 2025-11-03T16:15:21.9962715Z * [new branch] gh/bobrenjc93/649/head -> origin/gh/bobrenjc93/649/head 2025-11-03T16:15:21.9962869Z * [new branch] gh/bobrenjc93/649/orig -> origin/gh/bobrenjc93/649/orig 2025-11-03T16:15:21.9967716Z * [new branch] gh/bobrenjc93/650/base -> origin/gh/bobrenjc93/650/base 2025-11-03T16:15:21.9968033Z * [new branch] gh/bobrenjc93/650/head -> origin/gh/bobrenjc93/650/head 2025-11-03T16:15:21.9968268Z * [new branch] gh/bobrenjc93/650/orig -> origin/gh/bobrenjc93/650/orig 2025-11-03T16:15:21.9968559Z * [new branch] gh/bobrenjc93/651/base -> origin/gh/bobrenjc93/651/base 2025-11-03T16:15:21.9968834Z * [new branch] gh/bobrenjc93/651/head -> origin/gh/bobrenjc93/651/head 2025-11-03T16:15:21.9968984Z * [new branch] gh/bobrenjc93/651/orig -> origin/gh/bobrenjc93/651/orig 2025-11-03T16:15:21.9969128Z * [new branch] gh/bobrenjc93/652/base -> origin/gh/bobrenjc93/652/base 2025-11-03T16:15:21.9969269Z * [new branch] gh/bobrenjc93/652/head -> origin/gh/bobrenjc93/652/head 2025-11-03T16:15:21.9969405Z * [new branch] gh/bobrenjc93/652/orig -> origin/gh/bobrenjc93/652/orig 2025-11-03T16:15:21.9971317Z * [new branch] gh/bobrenjc93/653/base -> origin/gh/bobrenjc93/653/base 2025-11-03T16:15:21.9971491Z * [new branch] gh/bobrenjc93/653/head -> origin/gh/bobrenjc93/653/head 2025-11-03T16:15:21.9971638Z * [new branch] gh/bobrenjc93/653/orig -> origin/gh/bobrenjc93/653/orig 2025-11-03T16:15:21.9973624Z * [new branch] gh/bobrenjc93/654/base -> origin/gh/bobrenjc93/654/base 2025-11-03T16:15:21.9973803Z * [new branch] gh/bobrenjc93/654/head -> origin/gh/bobrenjc93/654/head 2025-11-03T16:15:21.9973957Z * [new branch] gh/bobrenjc93/654/orig -> origin/gh/bobrenjc93/654/orig 2025-11-03T16:15:21.9974457Z * [new branch] gh/bobrenjc93/655/base -> origin/gh/bobrenjc93/655/base 2025-11-03T16:15:21.9975912Z * [new branch] gh/bobrenjc93/655/head -> origin/gh/bobrenjc93/655/head 2025-11-03T16:15:21.9976104Z * [new branch] gh/bobrenjc93/655/orig -> origin/gh/bobrenjc93/655/orig 2025-11-03T16:15:21.9978720Z * [new branch] gh/bobrenjc93/656/base -> origin/gh/bobrenjc93/656/base 2025-11-03T16:15:21.9978894Z * [new branch] gh/bobrenjc93/656/head -> origin/gh/bobrenjc93/656/head 2025-11-03T16:15:21.9979036Z * [new branch] gh/bobrenjc93/656/orig -> origin/gh/bobrenjc93/656/orig 2025-11-03T16:15:21.9979205Z * [new branch] gh/bobrenjc93/657/base -> origin/gh/bobrenjc93/657/base 2025-11-03T16:15:21.9979621Z * [new branch] gh/bobrenjc93/657/head -> origin/gh/bobrenjc93/657/head 2025-11-03T16:15:21.9980322Z * [new branch] gh/bobrenjc93/657/orig -> origin/gh/bobrenjc93/657/orig 2025-11-03T16:15:21.9985029Z * [new branch] gh/bobrenjc93/658/base -> origin/gh/bobrenjc93/658/base 2025-11-03T16:15:21.9985362Z * [new branch] gh/bobrenjc93/658/head -> origin/gh/bobrenjc93/658/head 2025-11-03T16:15:21.9985591Z * [new branch] gh/bobrenjc93/658/orig -> origin/gh/bobrenjc93/658/orig 2025-11-03T16:15:21.9985755Z * [new branch] gh/bobrenjc93/659/base -> origin/gh/bobrenjc93/659/base 2025-11-03T16:15:21.9985973Z * [new branch] gh/bobrenjc93/659/head -> origin/gh/bobrenjc93/659/head 2025-11-03T16:15:21.9986211Z * [new branch] gh/bobrenjc93/659/orig -> origin/gh/bobrenjc93/659/orig 2025-11-03T16:15:21.9986836Z * [new branch] gh/bobrenjc93/660/base -> origin/gh/bobrenjc93/660/base 2025-11-03T16:15:21.9987010Z * [new branch] gh/bobrenjc93/660/head -> origin/gh/bobrenjc93/660/head 2025-11-03T16:15:21.9987330Z * [new branch] gh/bobrenjc93/660/orig -> origin/gh/bobrenjc93/660/orig 2025-11-03T16:15:21.9988629Z * [new branch] gh/bobrenjc93/661/base -> origin/gh/bobrenjc93/661/base 2025-11-03T16:15:21.9988983Z * [new branch] gh/bobrenjc93/661/head -> origin/gh/bobrenjc93/661/head 2025-11-03T16:15:21.9990111Z * [new branch] gh/bobrenjc93/661/orig -> origin/gh/bobrenjc93/661/orig 2025-11-03T16:15:21.9990693Z * [new branch] gh/bobrenjc93/662/base -> origin/gh/bobrenjc93/662/base 2025-11-03T16:15:21.9991415Z * [new branch] gh/bobrenjc93/662/head -> origin/gh/bobrenjc93/662/head 2025-11-03T16:15:21.9991971Z * [new branch] gh/bobrenjc93/662/orig -> origin/gh/bobrenjc93/662/orig 2025-11-03T16:15:21.9993095Z * [new branch] gh/bobrenjc93/663/base -> origin/gh/bobrenjc93/663/base 2025-11-03T16:15:21.9993380Z * [new branch] gh/bobrenjc93/663/head -> origin/gh/bobrenjc93/663/head 2025-11-03T16:15:21.9994352Z * [new branch] gh/bobrenjc93/663/orig -> origin/gh/bobrenjc93/663/orig 2025-11-03T16:15:21.9997923Z * [new branch] gh/bobrenjc93/664/base -> origin/gh/bobrenjc93/664/base 2025-11-03T16:15:21.9998098Z * [new branch] gh/bobrenjc93/664/head -> origin/gh/bobrenjc93/664/head 2025-11-03T16:15:21.9998246Z * [new branch] gh/bobrenjc93/664/orig -> origin/gh/bobrenjc93/664/orig 2025-11-03T16:15:21.9998385Z * [new branch] gh/bobrenjc93/665/base -> origin/gh/bobrenjc93/665/base 2025-11-03T16:15:21.9998542Z * [new branch] gh/bobrenjc93/665/head -> origin/gh/bobrenjc93/665/head 2025-11-03T16:15:21.9998686Z * [new branch] gh/bobrenjc93/665/orig -> origin/gh/bobrenjc93/665/orig 2025-11-03T16:15:21.9999453Z * [new branch] gh/bobrenjc93/666/base -> origin/gh/bobrenjc93/666/base 2025-11-03T16:15:22.0000089Z * [new branch] gh/bobrenjc93/666/head -> origin/gh/bobrenjc93/666/head 2025-11-03T16:15:22.0000729Z * [new branch] gh/bobrenjc93/666/orig -> origin/gh/bobrenjc93/666/orig 2025-11-03T16:15:22.0002124Z * [new branch] gh/bobrenjc93/667/base -> origin/gh/bobrenjc93/667/base 2025-11-03T16:15:22.0002270Z * [new branch] gh/bobrenjc93/667/head -> origin/gh/bobrenjc93/667/head 2025-11-03T16:15:22.0002710Z * [new branch] gh/bobrenjc93/667/orig -> origin/gh/bobrenjc93/667/orig 2025-11-03T16:15:22.0004369Z * [new branch] gh/bobrenjc93/668/base -> origin/gh/bobrenjc93/668/base 2025-11-03T16:15:22.0004745Z * [new branch] gh/bobrenjc93/668/head -> origin/gh/bobrenjc93/668/head 2025-11-03T16:15:22.0005028Z * [new branch] gh/bobrenjc93/668/orig -> origin/gh/bobrenjc93/668/orig 2025-11-03T16:15:22.0005740Z * [new branch] gh/bobrenjc93/669/base -> origin/gh/bobrenjc93/669/base 2025-11-03T16:15:22.0007122Z * [new branch] gh/bobrenjc93/669/head -> origin/gh/bobrenjc93/669/head 2025-11-03T16:15:22.0007305Z * [new branch] gh/bobrenjc93/669/orig -> origin/gh/bobrenjc93/669/orig 2025-11-03T16:15:22.0008230Z * [new branch] gh/bobrenjc93/670/base -> origin/gh/bobrenjc93/670/base 2025-11-03T16:15:22.0008493Z * [new branch] gh/bobrenjc93/670/head -> origin/gh/bobrenjc93/670/head 2025-11-03T16:15:22.0009385Z * [new branch] gh/bobrenjc93/670/orig -> origin/gh/bobrenjc93/670/orig 2025-11-03T16:15:22.0011891Z * [new branch] gh/bobrenjc93/671/base -> origin/gh/bobrenjc93/671/base 2025-11-03T16:15:22.0012253Z * [new branch] gh/bobrenjc93/671/head -> origin/gh/bobrenjc93/671/head 2025-11-03T16:15:22.0012515Z * [new branch] gh/bobrenjc93/671/orig -> origin/gh/bobrenjc93/671/orig 2025-11-03T16:15:22.0012692Z * [new branch] gh/bobrenjc93/672/base -> origin/gh/bobrenjc93/672/base 2025-11-03T16:15:22.0012985Z * [new branch] gh/bobrenjc93/672/head -> origin/gh/bobrenjc93/672/head 2025-11-03T16:15:22.0013710Z * [new branch] gh/bobrenjc93/672/orig -> origin/gh/bobrenjc93/672/orig 2025-11-03T16:15:22.0014684Z * [new branch] gh/bobrenjc93/673/base -> origin/gh/bobrenjc93/673/base 2025-11-03T16:15:22.0015012Z * [new branch] gh/bobrenjc93/673/head -> origin/gh/bobrenjc93/673/head 2025-11-03T16:15:22.0017684Z * [new branch] gh/bobrenjc93/673/orig -> origin/gh/bobrenjc93/673/orig 2025-11-03T16:15:22.0018233Z * [new branch] gh/bobrenjc93/674/base -> origin/gh/bobrenjc93/674/base 2025-11-03T16:15:22.0018512Z * [new branch] gh/bobrenjc93/674/head -> origin/gh/bobrenjc93/674/head 2025-11-03T16:15:22.0019184Z * [new branch] gh/bobrenjc93/674/orig -> origin/gh/bobrenjc93/674/orig 2025-11-03T16:15:22.0019372Z * [new branch] gh/bobrenjc93/675/base -> origin/gh/bobrenjc93/675/base 2025-11-03T16:15:22.0019519Z * [new branch] gh/bobrenjc93/675/head -> origin/gh/bobrenjc93/675/head 2025-11-03T16:15:22.0019835Z * [new branch] gh/bobrenjc93/675/orig -> origin/gh/bobrenjc93/675/orig 2025-11-03T16:15:22.0020756Z * [new branch] gh/bobrenjc93/676/base -> origin/gh/bobrenjc93/676/base 2025-11-03T16:15:22.0021075Z * [new branch] gh/bobrenjc93/676/head -> origin/gh/bobrenjc93/676/head 2025-11-03T16:15:22.0023825Z * [new branch] gh/bobrenjc93/676/orig -> origin/gh/bobrenjc93/676/orig 2025-11-03T16:15:22.0024175Z * [new branch] gh/bobrenjc93/677/base -> origin/gh/bobrenjc93/677/base 2025-11-03T16:15:22.0024422Z * [new branch] gh/bobrenjc93/677/head -> origin/gh/bobrenjc93/677/head 2025-11-03T16:15:22.0024579Z * [new branch] gh/bobrenjc93/677/orig -> origin/gh/bobrenjc93/677/orig 2025-11-03T16:15:22.0024823Z * [new branch] gh/bobrenjc93/678/base -> origin/gh/bobrenjc93/678/base 2025-11-03T16:15:22.0025166Z * [new branch] gh/bobrenjc93/678/head -> origin/gh/bobrenjc93/678/head 2025-11-03T16:15:22.0026104Z * [new branch] gh/bobrenjc93/678/orig -> origin/gh/bobrenjc93/678/orig 2025-11-03T16:15:22.0028335Z * [new branch] gh/bobrenjc93/679/base -> origin/gh/bobrenjc93/679/base 2025-11-03T16:15:22.0028660Z * [new branch] gh/bobrenjc93/679/head -> origin/gh/bobrenjc93/679/head 2025-11-03T16:15:22.0028896Z * [new branch] gh/bobrenjc93/679/orig -> origin/gh/bobrenjc93/679/orig 2025-11-03T16:15:22.0029394Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-11-03T16:15:22.0030402Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-11-03T16:15:22.0030984Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-11-03T16:15:22.0032594Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-11-03T16:15:22.0032751Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-11-03T16:15:22.0032919Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-11-03T16:15:22.0035182Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-11-03T16:15:22.0035477Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-11-03T16:15:22.0035710Z * [new branch] gh/c00w/55/base -> origin/gh/c00w/55/base 2025-11-03T16:15:22.0035933Z * [new branch] gh/c00w/55/head -> origin/gh/c00w/55/head 2025-11-03T16:15:22.0036341Z * [new branch] gh/c00w/55/orig -> origin/gh/c00w/55/orig 2025-11-03T16:15:22.0037739Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-11-03T16:15:22.0037916Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-11-03T16:15:22.0039955Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-11-03T16:15:22.0040258Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-11-03T16:15:22.0040397Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-11-03T16:15:22.0040529Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-11-03T16:15:22.0044712Z * [new branch] gh/c00w/58/base -> origin/gh/c00w/58/base 2025-11-03T16:15:22.0045153Z * [new branch] gh/c00w/58/head -> origin/gh/c00w/58/head 2025-11-03T16:15:22.0045408Z * [new branch] gh/c00w/58/orig -> origin/gh/c00w/58/orig 2025-11-03T16:15:22.0045661Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-11-03T16:15:22.0046265Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-11-03T16:15:22.0046421Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-11-03T16:15:22.0046582Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-11-03T16:15:22.0046950Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-11-03T16:15:22.0048846Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-11-03T16:15:22.0049207Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-11-03T16:15:22.0049465Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-11-03T16:15:22.0054147Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-11-03T16:15:22.0054479Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-11-03T16:15:22.0054717Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-11-03T16:15:22.0054876Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-11-03T16:15:22.0055130Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-11-03T16:15:22.0055281Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-11-03T16:15:22.0055960Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-11-03T16:15:22.0056160Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-11-03T16:15:22.0056342Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-11-03T16:15:22.0057635Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-11-03T16:15:22.0057921Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-11-03T16:15:22.0058416Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-11-03T16:15:22.0060415Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-11-03T16:15:22.0060765Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-11-03T16:15:22.0060993Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-11-03T16:15:22.0061422Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-11-03T16:15:22.0061845Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-11-03T16:15:22.0063163Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-11-03T16:15:22.0063537Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-11-03T16:15:22.0065825Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-11-03T16:15:22.0066150Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-11-03T16:15:22.0066378Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-11-03T16:15:22.0066544Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-11-03T16:15:22.0067701Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-11-03T16:15:22.0068229Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-11-03T16:15:22.0068851Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-11-03T16:15:22.0069994Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-11-03T16:15:22.0070498Z * [new branch] gh/coconutruben/81/base -> origin/gh/coconutruben/81/base 2025-11-03T16:15:22.0071247Z * [new branch] gh/coconutruben/81/head -> origin/gh/coconutruben/81/head 2025-11-03T16:15:22.0071912Z * [new branch] gh/coconutruben/81/orig -> origin/gh/coconutruben/81/orig 2025-11-03T16:15:22.0073082Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-11-03T16:15:22.0073236Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-11-03T16:15:22.0075472Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-11-03T16:15:22.0075836Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-11-03T16:15:22.0076093Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-11-03T16:15:22.0076636Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-11-03T16:15:22.0080898Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-11-03T16:15:22.0081239Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-11-03T16:15:22.0081472Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-11-03T16:15:22.0081641Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-11-03T16:15:22.0081779Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-11-03T16:15:22.0082074Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-11-03T16:15:22.0082703Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-11-03T16:15:22.0082907Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-11-03T16:15:22.0087588Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-11-03T16:15:22.0087926Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-11-03T16:15:22.0088197Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-11-03T16:15:22.0088432Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-11-03T16:15:22.0088601Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-11-03T16:15:22.0088855Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-11-03T16:15:22.0089510Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-11-03T16:15:22.0089875Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-11-03T16:15:22.0090188Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-11-03T16:15:22.0094413Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-11-03T16:15:22.0094728Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-11-03T16:15:22.0094962Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-11-03T16:15:22.0095213Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-11-03T16:15:22.0095443Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-11-03T16:15:22.0095746Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-11-03T16:15:22.0095899Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-11-03T16:15:22.0096293Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-11-03T16:15:22.0097240Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-11-03T16:15:22.0100737Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-11-03T16:15:22.0101066Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-11-03T16:15:22.0101306Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-11-03T16:15:22.0101464Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-11-03T16:15:22.0101604Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-11-03T16:15:22.0101900Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-11-03T16:15:22.0102520Z * [new branch] gh/desertfire/600/base -> origin/gh/desertfire/600/base 2025-11-03T16:15:22.0102826Z * [new branch] gh/desertfire/600/head -> origin/gh/desertfire/600/head 2025-11-03T16:15:22.0105706Z * [new branch] gh/desertfire/600/orig -> origin/gh/desertfire/600/orig 2025-11-03T16:15:22.0106025Z * [new branch] gh/desertfire/601/base -> origin/gh/desertfire/601/base 2025-11-03T16:15:22.0106243Z * [new branch] gh/desertfire/601/head -> origin/gh/desertfire/601/head 2025-11-03T16:15:22.0106397Z * [new branch] gh/desertfire/601/orig -> origin/gh/desertfire/601/orig 2025-11-03T16:15:22.0106616Z * [new branch] gh/desertfire/602/base -> origin/gh/desertfire/602/base 2025-11-03T16:15:22.0107101Z * [new branch] gh/desertfire/602/head -> origin/gh/desertfire/602/head 2025-11-03T16:15:22.0108084Z * [new branch] gh/desertfire/602/orig -> origin/gh/desertfire/602/orig 2025-11-03T16:15:22.0108591Z * [new branch] gh/desertfire/603/base -> origin/gh/desertfire/603/base 2025-11-03T16:15:22.0109451Z * [new branch] gh/desertfire/603/head -> origin/gh/desertfire/603/head 2025-11-03T16:15:22.0109816Z * [new branch] gh/desertfire/603/orig -> origin/gh/desertfire/603/orig 2025-11-03T16:15:22.0110930Z * [new branch] gh/desertfire/604/base -> origin/gh/desertfire/604/base 2025-11-03T16:15:22.0111486Z * [new branch] gh/desertfire/604/head -> origin/gh/desertfire/604/head 2025-11-03T16:15:22.0112159Z * [new branch] gh/desertfire/604/orig -> origin/gh/desertfire/604/orig 2025-11-03T16:15:22.0113410Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-11-03T16:15:22.0113761Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-11-03T16:15:22.0118345Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-11-03T16:15:22.0120352Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-11-03T16:15:22.0120624Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-11-03T16:15:22.0125713Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-11-03T16:15:22.0127657Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-11-03T16:15:22.0127965Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-11-03T16:15:22.0128185Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-11-03T16:15:22.0128339Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-11-03T16:15:22.0128673Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-11-03T16:15:22.0128947Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-11-03T16:15:22.0129086Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-11-03T16:15:22.0129338Z * [new branch] gh/drisspg/187/base -> origin/gh/drisspg/187/base 2025-11-03T16:15:22.0129487Z * [new branch] gh/drisspg/187/head -> origin/gh/drisspg/187/head 2025-11-03T16:15:22.0129685Z * [new branch] gh/drisspg/187/orig -> origin/gh/drisspg/187/orig 2025-11-03T16:15:22.0130389Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-11-03T16:15:22.0130704Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-11-03T16:15:22.0130929Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-11-03T16:15:22.0131214Z * [new branch] gh/drisspg/197/base -> origin/gh/drisspg/197/base 2025-11-03T16:15:22.0131365Z * [new branch] gh/drisspg/197/head -> origin/gh/drisspg/197/head 2025-11-03T16:15:22.0131614Z * [new branch] gh/drisspg/197/orig -> origin/gh/drisspg/197/orig 2025-11-03T16:15:22.0132266Z * [new branch] gh/drisspg/199/base -> origin/gh/drisspg/199/base 2025-11-03T16:15:22.0132446Z * [new branch] gh/drisspg/199/head -> origin/gh/drisspg/199/head 2025-11-03T16:15:22.0132585Z * [new branch] gh/drisspg/199/orig -> origin/gh/drisspg/199/orig 2025-11-03T16:15:22.0132713Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-11-03T16:15:22.0132842Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-11-03T16:15:22.0132986Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-11-03T16:15:22.0133370Z * [new branch] gh/drisspg/210/base -> origin/gh/drisspg/210/base 2025-11-03T16:15:22.0134386Z * [new branch] gh/drisspg/210/head -> origin/gh/drisspg/210/head 2025-11-03T16:15:22.0134747Z * [new branch] gh/drisspg/210/orig -> origin/gh/drisspg/210/orig 2025-11-03T16:15:22.0136884Z * [new branch] gh/drisspg/211/base -> origin/gh/drisspg/211/base 2025-11-03T16:15:22.0137194Z * [new branch] gh/drisspg/211/head -> origin/gh/drisspg/211/head 2025-11-03T16:15:22.0137351Z * [new branch] gh/drisspg/211/orig -> origin/gh/drisspg/211/orig 2025-11-03T16:15:22.0137485Z * [new branch] gh/drisspg/212/base -> origin/gh/drisspg/212/base 2025-11-03T16:15:22.0138444Z * [new branch] gh/drisspg/212/head -> origin/gh/drisspg/212/head 2025-11-03T16:15:22.0138818Z * [new branch] gh/drisspg/212/orig -> origin/gh/drisspg/212/orig 2025-11-03T16:15:22.0140892Z * [new branch] gh/drisspg/213/base -> origin/gh/drisspg/213/base 2025-11-03T16:15:22.0141208Z * [new branch] gh/drisspg/213/head -> origin/gh/drisspg/213/head 2025-11-03T16:15:22.0141444Z * [new branch] gh/drisspg/213/orig -> origin/gh/drisspg/213/orig 2025-11-03T16:15:22.0141695Z * [new branch] gh/drisspg/214/base -> origin/gh/drisspg/214/base 2025-11-03T16:15:22.0143835Z * [new branch] gh/drisspg/214/head -> origin/gh/drisspg/214/head 2025-11-03T16:15:22.0144152Z * [new branch] gh/drisspg/214/orig -> origin/gh/drisspg/214/orig 2025-11-03T16:15:22.0144363Z * [new branch] gh/drisspg/215/base -> origin/gh/drisspg/215/base 2025-11-03T16:15:22.0144515Z * [new branch] gh/drisspg/215/head -> origin/gh/drisspg/215/head 2025-11-03T16:15:22.0145177Z * [new branch] gh/drisspg/215/orig -> origin/gh/drisspg/215/orig 2025-11-03T16:15:22.0149089Z * [new branch] gh/drisspg/216/base -> origin/gh/drisspg/216/base 2025-11-03T16:15:22.0149406Z * [new branch] gh/drisspg/216/head -> origin/gh/drisspg/216/head 2025-11-03T16:15:22.0149646Z * [new branch] gh/drisspg/216/orig -> origin/gh/drisspg/216/orig 2025-11-03T16:15:22.0149791Z * [new branch] gh/drisspg/217/base -> origin/gh/drisspg/217/base 2025-11-03T16:15:22.0150031Z * [new branch] gh/drisspg/217/head -> origin/gh/drisspg/217/head 2025-11-03T16:15:22.0150174Z * [new branch] gh/drisspg/217/orig -> origin/gh/drisspg/217/orig 2025-11-03T16:15:22.0150791Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-11-03T16:15:22.0151877Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-11-03T16:15:22.0154373Z * [new branch] gh/dzmitry-huba/1/base -> origin/gh/dzmitry-huba/1/base 2025-11-03T16:15:22.0154552Z * [new branch] gh/dzmitry-huba/1/head -> origin/gh/dzmitry-huba/1/head 2025-11-03T16:15:22.0154716Z * [new branch] gh/dzmitry-huba/10/base -> origin/gh/dzmitry-huba/10/base 2025-11-03T16:15:22.0154905Z * [new branch] gh/dzmitry-huba/10/head -> origin/gh/dzmitry-huba/10/head 2025-11-03T16:15:22.0157907Z * [new branch] gh/dzmitry-huba/10/orig -> origin/gh/dzmitry-huba/10/orig 2025-11-03T16:15:22.0158237Z * [new branch] gh/dzmitry-huba/11/base -> origin/gh/dzmitry-huba/11/base 2025-11-03T16:15:22.0158461Z * [new branch] gh/dzmitry-huba/11/head -> origin/gh/dzmitry-huba/11/head 2025-11-03T16:15:22.0158620Z * [new branch] gh/dzmitry-huba/11/orig -> origin/gh/dzmitry-huba/11/orig 2025-11-03T16:15:22.0158869Z * [new branch] gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base 2025-11-03T16:15:22.0159782Z * [new branch] gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head 2025-11-03T16:15:22.0160135Z * [new branch] gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig 2025-11-03T16:15:22.0163122Z * [new branch] gh/dzmitry-huba/2/base -> origin/gh/dzmitry-huba/2/base 2025-11-03T16:15:22.0163464Z * [new branch] gh/dzmitry-huba/2/head -> origin/gh/dzmitry-huba/2/head 2025-11-03T16:15:22.0163717Z * [new branch] gh/dzmitry-huba/3/base -> origin/gh/dzmitry-huba/3/base 2025-11-03T16:15:22.0163946Z * [new branch] gh/dzmitry-huba/3/head -> origin/gh/dzmitry-huba/3/head 2025-11-03T16:15:22.0164581Z * [new branch] gh/dzmitry-huba/4/base -> origin/gh/dzmitry-huba/4/base 2025-11-03T16:15:22.0164762Z * [new branch] gh/dzmitry-huba/4/head -> origin/gh/dzmitry-huba/4/head 2025-11-03T16:15:22.0165064Z * [new branch] gh/dzmitry-huba/4/orig -> origin/gh/dzmitry-huba/4/orig 2025-11-03T16:15:22.0166470Z * [new branch] gh/dzmitry-huba/5/base -> origin/gh/dzmitry-huba/5/base 2025-11-03T16:15:22.0166716Z * [new branch] gh/dzmitry-huba/5/head -> origin/gh/dzmitry-huba/5/head 2025-11-03T16:15:22.0168633Z * [new branch] gh/dzmitry-huba/5/orig -> origin/gh/dzmitry-huba/5/orig 2025-11-03T16:15:22.0168818Z * [new branch] gh/dzmitry-huba/6/base -> origin/gh/dzmitry-huba/6/base 2025-11-03T16:15:22.0168973Z * [new branch] gh/dzmitry-huba/6/head -> origin/gh/dzmitry-huba/6/head 2025-11-03T16:15:22.0169655Z * [new branch] gh/dzmitry-huba/6/orig -> origin/gh/dzmitry-huba/6/orig 2025-11-03T16:15:22.0172878Z * [new branch] gh/dzmitry-huba/7/base -> origin/gh/dzmitry-huba/7/base 2025-11-03T16:15:22.0173056Z * [new branch] gh/dzmitry-huba/7/head -> origin/gh/dzmitry-huba/7/head 2025-11-03T16:15:22.0173363Z * [new branch] gh/dzmitry-huba/7/orig -> origin/gh/dzmitry-huba/7/orig 2025-11-03T16:15:22.0173503Z * [new branch] gh/dzmitry-huba/8/base -> origin/gh/dzmitry-huba/8/base 2025-11-03T16:15:22.0173838Z * [new branch] gh/dzmitry-huba/8/head -> origin/gh/dzmitry-huba/8/head 2025-11-03T16:15:22.0174064Z * [new branch] gh/dzmitry-huba/8/orig -> origin/gh/dzmitry-huba/8/orig 2025-11-03T16:15:22.0177757Z * [new branch] gh/dzmitry-huba/9/base -> origin/gh/dzmitry-huba/9/base 2025-11-03T16:15:22.0178090Z * [new branch] gh/dzmitry-huba/9/head -> origin/gh/dzmitry-huba/9/head 2025-11-03T16:15:22.0178325Z * [new branch] gh/dzmitry-huba/9/orig -> origin/gh/dzmitry-huba/9/orig 2025-11-03T16:15:22.0178491Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-11-03T16:15:22.0178754Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-11-03T16:15:22.0179382Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-11-03T16:15:22.0179861Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-11-03T16:15:22.0180324Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-11-03T16:15:22.0181083Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-11-03T16:15:22.0183476Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-11-03T16:15:22.0183803Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-11-03T16:15:22.0184045Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-11-03T16:15:22.0184194Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-11-03T16:15:22.0184652Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-11-03T16:15:22.0185636Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-11-03T16:15:22.0190064Z * [new branch] gh/eellison/836/base -> origin/gh/eellison/836/base 2025-11-03T16:15:22.0190399Z * [new branch] gh/eellison/836/head -> origin/gh/eellison/836/head 2025-11-03T16:15:22.0190540Z * [new branch] gh/eellison/836/orig -> origin/gh/eellison/836/orig 2025-11-03T16:15:22.0190669Z * [new branch] gh/eellison/837/base -> origin/gh/eellison/837/base 2025-11-03T16:15:22.0190800Z * [new branch] gh/eellison/837/head -> origin/gh/eellison/837/head 2025-11-03T16:15:22.0190938Z * [new branch] gh/eellison/837/orig -> origin/gh/eellison/837/orig 2025-11-03T16:15:22.0191344Z * [new branch] gh/eellison/838/base -> origin/gh/eellison/838/base 2025-11-03T16:15:22.0191538Z * [new branch] gh/eellison/838/head -> origin/gh/eellison/838/head 2025-11-03T16:15:22.0192414Z * [new branch] gh/eellison/838/orig -> origin/gh/eellison/838/orig 2025-11-03T16:15:22.0195339Z * [new branch] gh/eellison/839/base -> origin/gh/eellison/839/base 2025-11-03T16:15:22.0195512Z * [new branch] gh/eellison/839/head -> origin/gh/eellison/839/head 2025-11-03T16:15:22.0195642Z * [new branch] gh/eellison/839/orig -> origin/gh/eellison/839/orig 2025-11-03T16:15:22.0195776Z * [new branch] gh/eellison/840/base -> origin/gh/eellison/840/base 2025-11-03T16:15:22.0196055Z * [new branch] gh/eellison/840/head -> origin/gh/eellison/840/head 2025-11-03T16:15:22.0197653Z * [new branch] gh/eellison/840/orig -> origin/gh/eellison/840/orig 2025-11-03T16:15:22.0197816Z * [new branch] gh/eellison/841/base -> origin/gh/eellison/841/base 2025-11-03T16:15:22.0198304Z * [new branch] gh/eellison/841/head -> origin/gh/eellison/841/head 2025-11-03T16:15:22.0198801Z * [new branch] gh/eellison/841/orig -> origin/gh/eellison/841/orig 2025-11-03T16:15:22.0200698Z * [new branch] gh/eellison/842/base -> origin/gh/eellison/842/base 2025-11-03T16:15:22.0201035Z * [new branch] gh/eellison/842/head -> origin/gh/eellison/842/head 2025-11-03T16:15:22.0201269Z * [new branch] gh/eellison/842/orig -> origin/gh/eellison/842/orig 2025-11-03T16:15:22.0202950Z * [new branch] gh/eellison/843/base -> origin/gh/eellison/843/base 2025-11-03T16:15:22.0203283Z * [new branch] gh/eellison/843/head -> origin/gh/eellison/843/head 2025-11-03T16:15:22.0203502Z * [new branch] gh/eellison/843/orig -> origin/gh/eellison/843/orig 2025-11-03T16:15:22.0205418Z * [new branch] gh/eellison/844/base -> origin/gh/eellison/844/base 2025-11-03T16:15:22.0205755Z * [new branch] gh/eellison/844/head -> origin/gh/eellison/844/head 2025-11-03T16:15:22.0205997Z * [new branch] gh/eellison/844/orig -> origin/gh/eellison/844/orig 2025-11-03T16:15:22.0206518Z * [new branch] gh/eellison/845/base -> origin/gh/eellison/845/base 2025-11-03T16:15:22.0208798Z * [new branch] gh/eellison/845/head -> origin/gh/eellison/845/head 2025-11-03T16:15:22.0209114Z * [new branch] gh/eellison/845/orig -> origin/gh/eellison/845/orig 2025-11-03T16:15:22.0209340Z * [new branch] gh/eellison/846/base -> origin/gh/eellison/846/base 2025-11-03T16:15:22.0209491Z * [new branch] gh/eellison/846/head -> origin/gh/eellison/846/head 2025-11-03T16:15:22.0209946Z * [new branch] gh/eellison/846/orig -> origin/gh/eellison/846/orig 2025-11-03T16:15:22.0213989Z * [new branch] gh/eellison/847/base -> origin/gh/eellison/847/base 2025-11-03T16:15:22.0214305Z * [new branch] gh/eellison/847/head -> origin/gh/eellison/847/head 2025-11-03T16:15:22.0214525Z * [new branch] gh/eellison/847/orig -> origin/gh/eellison/847/orig 2025-11-03T16:15:22.0215026Z * [new branch] gh/eellison/848/base -> origin/gh/eellison/848/base 2025-11-03T16:15:22.0215620Z * [new branch] gh/eellison/848/head -> origin/gh/eellison/848/head 2025-11-03T16:15:22.0215786Z * [new branch] gh/eellison/848/orig -> origin/gh/eellison/848/orig 2025-11-03T16:15:22.0215927Z * [new branch] gh/eellison/849/base -> origin/gh/eellison/849/base 2025-11-03T16:15:22.0216309Z * [new branch] gh/eellison/849/head -> origin/gh/eellison/849/head 2025-11-03T16:15:22.0217529Z * [new branch] gh/eellison/849/orig -> origin/gh/eellison/849/orig 2025-11-03T16:15:22.0217944Z * [new branch] gh/eellison/850/base -> origin/gh/eellison/850/base 2025-11-03T16:15:22.0220578Z * [new branch] gh/eellison/850/head -> origin/gh/eellison/850/head 2025-11-03T16:15:22.0220915Z * [new branch] gh/eellison/850/orig -> origin/gh/eellison/850/orig 2025-11-03T16:15:22.0221146Z * [new branch] gh/eellison/851/base -> origin/gh/eellison/851/base 2025-11-03T16:15:22.0221304Z * [new branch] gh/eellison/851/head -> origin/gh/eellison/851/head 2025-11-03T16:15:22.0221523Z * [new branch] gh/eellison/851/orig -> origin/gh/eellison/851/orig 2025-11-03T16:15:22.0222095Z * [new branch] gh/eellison/852/base -> origin/gh/eellison/852/base 2025-11-03T16:15:22.0222935Z * [new branch] gh/eellison/852/head -> origin/gh/eellison/852/head 2025-11-03T16:15:22.0223266Z * [new branch] gh/eellison/852/orig -> origin/gh/eellison/852/orig 2025-11-03T16:15:22.0226011Z * [new branch] gh/eellison/853/base -> origin/gh/eellison/853/base 2025-11-03T16:15:22.0226364Z * [new branch] gh/eellison/853/head -> origin/gh/eellison/853/head 2025-11-03T16:15:22.0226620Z * [new branch] gh/eellison/853/orig -> origin/gh/eellison/853/orig 2025-11-03T16:15:22.0226775Z * [new branch] gh/eellison/854/base -> origin/gh/eellison/854/base 2025-11-03T16:15:22.0227231Z * [new branch] gh/eellison/854/head -> origin/gh/eellison/854/head 2025-11-03T16:15:22.0227898Z * [new branch] gh/eellison/854/orig -> origin/gh/eellison/854/orig 2025-11-03T16:15:22.0228846Z * [new branch] gh/eellison/855/base -> origin/gh/eellison/855/base 2025-11-03T16:15:22.0229072Z * [new branch] gh/eellison/855/head -> origin/gh/eellison/855/head 2025-11-03T16:15:22.0230426Z * [new branch] gh/eellison/855/orig -> origin/gh/eellison/855/orig 2025-11-03T16:15:22.0230655Z * [new branch] gh/eellison/856/base -> origin/gh/eellison/856/base 2025-11-03T16:15:22.0232795Z * [new branch] gh/eellison/856/head -> origin/gh/eellison/856/head 2025-11-03T16:15:22.0232974Z * [new branch] gh/eellison/856/orig -> origin/gh/eellison/856/orig 2025-11-03T16:15:22.0233111Z * [new branch] gh/eellison/857/base -> origin/gh/eellison/857/base 2025-11-03T16:15:22.0233426Z * [new branch] gh/eellison/857/head -> origin/gh/eellison/857/head 2025-11-03T16:15:22.0235120Z * [new branch] gh/eellison/857/orig -> origin/gh/eellison/857/orig 2025-11-03T16:15:22.0235442Z * [new branch] gh/eellison/858/base -> origin/gh/eellison/858/base 2025-11-03T16:15:22.0235748Z * [new branch] gh/eellison/858/head -> origin/gh/eellison/858/head 2025-11-03T16:15:22.0238212Z * [new branch] gh/eellison/858/orig -> origin/gh/eellison/858/orig 2025-11-03T16:15:22.0238549Z * [new branch] gh/eellison/859/base -> origin/gh/eellison/859/base 2025-11-03T16:15:22.0238796Z * [new branch] gh/eellison/859/head -> origin/gh/eellison/859/head 2025-11-03T16:15:22.0239217Z * [new branch] gh/eellison/859/orig -> origin/gh/eellison/859/orig 2025-11-03T16:15:22.0239952Z * [new branch] gh/eellison/860/base -> origin/gh/eellison/860/base 2025-11-03T16:15:22.0240139Z * [new branch] gh/eellison/860/head -> origin/gh/eellison/860/head 2025-11-03T16:15:22.0243222Z * [new branch] gh/eellison/860/orig -> origin/gh/eellison/860/orig 2025-11-03T16:15:22.0243539Z * [new branch] gh/eellison/861/base -> origin/gh/eellison/861/base 2025-11-03T16:15:22.0243770Z * [new branch] gh/eellison/861/head -> origin/gh/eellison/861/head 2025-11-03T16:15:22.0243931Z * [new branch] gh/eellison/861/orig -> origin/gh/eellison/861/orig 2025-11-03T16:15:22.0244172Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-11-03T16:15:22.0244510Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-11-03T16:15:22.0248139Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-11-03T16:15:22.0248443Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-11-03T16:15:22.0248662Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-11-03T16:15:22.0248801Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-11-03T16:15:22.0248928Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-11-03T16:15:22.0249176Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-11-03T16:15:22.0249893Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-11-03T16:15:22.0250366Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-11-03T16:15:22.0252701Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-11-03T16:15:22.0253015Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-11-03T16:15:22.0253226Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-11-03T16:15:22.0253357Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-11-03T16:15:22.0254266Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-11-03T16:15:22.0254657Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-11-03T16:15:22.0256494Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-11-03T16:15:22.0256820Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-11-03T16:15:22.0257021Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-11-03T16:15:22.0258625Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-11-03T16:15:22.0258965Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-11-03T16:15:22.0259170Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-11-03T16:15:22.0259771Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-11-03T16:15:22.0260957Z * [new branch] gh/etaf/162/base -> origin/gh/etaf/162/base 2025-11-03T16:15:22.0261219Z * [new branch] gh/etaf/162/head -> origin/gh/etaf/162/head 2025-11-03T16:15:22.0262822Z * [new branch] gh/etaf/162/orig -> origin/gh/etaf/162/orig 2025-11-03T16:15:22.0262986Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-11-03T16:15:22.0263519Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-11-03T16:15:22.0264084Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-11-03T16:15:22.0264993Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-11-03T16:15:22.0265739Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-11-03T16:15:22.0266175Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-11-03T16:15:22.0267337Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-11-03T16:15:22.0267564Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-11-03T16:15:22.0269336Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-11-03T16:15:22.0269519Z * [new branch] gh/etaf/171/base -> origin/gh/etaf/171/base 2025-11-03T16:15:22.0269785Z * [new branch] gh/etaf/171/head -> origin/gh/etaf/171/head 2025-11-03T16:15:22.0270901Z * [new branch] gh/etaf/171/orig -> origin/gh/etaf/171/orig 2025-11-03T16:15:22.0271405Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-11-03T16:15:22.0272266Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-11-03T16:15:22.0272813Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-11-03T16:15:22.0278491Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-11-03T16:15:22.0278680Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-11-03T16:15:22.0278838Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-11-03T16:15:22.0279128Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-11-03T16:15:22.0279278Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-11-03T16:15:22.0279438Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-11-03T16:15:22.0279581Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-11-03T16:15:22.0279899Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-11-03T16:15:22.0280505Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-11-03T16:15:22.0280982Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-11-03T16:15:22.0281970Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-11-03T16:15:22.0282458Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-11-03T16:15:22.0285456Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-11-03T16:15:22.0285779Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-11-03T16:15:22.0286014Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-11-03T16:15:22.0286169Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-11-03T16:15:22.0286416Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-11-03T16:15:22.0286738Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-11-03T16:15:22.0287710Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-11-03T16:15:22.0288035Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-11-03T16:15:22.0291434Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-11-03T16:15:22.0291829Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-11-03T16:15:22.0292141Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-11-03T16:15:22.0292608Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-11-03T16:15:22.0292899Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-11-03T16:15:22.0293467Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-11-03T16:15:22.0293636Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-11-03T16:15:22.0293941Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-11-03T16:15:22.0294270Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-11-03T16:15:22.0295472Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-11-03T16:15:22.0295819Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-11-03T16:15:22.0298718Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-11-03T16:15:22.0298905Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-11-03T16:15:22.0299035Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-11-03T16:15:22.0299162Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-11-03T16:15:22.0299408Z * [new branch] gh/ezyang/3148/base -> origin/gh/ezyang/3148/base 2025-11-03T16:15:22.0300985Z * [new branch] gh/ezyang/3148/head -> origin/gh/ezyang/3148/head 2025-11-03T16:15:22.0301298Z * [new branch] gh/ezyang/3148/orig -> origin/gh/ezyang/3148/orig 2025-11-03T16:15:22.0301955Z * [new branch] gh/ezyang/3158/base -> origin/gh/ezyang/3158/base 2025-11-03T16:15:22.0304500Z * [new branch] gh/ezyang/3158/head -> origin/gh/ezyang/3158/head 2025-11-03T16:15:22.0304831Z * [new branch] gh/ezyang/3158/orig -> origin/gh/ezyang/3158/orig 2025-11-03T16:15:22.0305078Z * [new branch] gh/ezyang/3165/base -> origin/gh/ezyang/3165/base 2025-11-03T16:15:22.0305232Z * [new branch] gh/ezyang/3165/head -> origin/gh/ezyang/3165/head 2025-11-03T16:15:22.0305357Z * [new branch] gh/ezyang/3165/orig -> origin/gh/ezyang/3165/orig 2025-11-03T16:15:22.0306785Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-11-03T16:15:22.0307096Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-11-03T16:15:22.0307312Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-11-03T16:15:22.0309601Z * [new branch] gh/ezyang/3168/base -> origin/gh/ezyang/3168/base 2025-11-03T16:15:22.0309769Z * [new branch] gh/ezyang/3168/head -> origin/gh/ezyang/3168/head 2025-11-03T16:15:22.0309912Z * [new branch] gh/ezyang/3168/orig -> origin/gh/ezyang/3168/orig 2025-11-03T16:15:22.0310803Z * [new branch] gh/ezyang/3169/base -> origin/gh/ezyang/3169/base 2025-11-03T16:15:22.0311244Z * [new branch] gh/ezyang/3169/head -> origin/gh/ezyang/3169/head 2025-11-03T16:15:22.0312160Z * [new branch] gh/ezyang/3169/orig -> origin/gh/ezyang/3169/orig 2025-11-03T16:15:22.0315893Z * [new branch] gh/ezyang/3170/base -> origin/gh/ezyang/3170/base 2025-11-03T16:15:22.0316440Z * [new branch] gh/ezyang/3170/head -> origin/gh/ezyang/3170/head 2025-11-03T16:15:22.0316585Z * [new branch] gh/ezyang/3170/orig -> origin/gh/ezyang/3170/orig 2025-11-03T16:15:22.0322312Z * [new branch] gh/ezyang/3171/base -> origin/gh/ezyang/3171/base 2025-11-03T16:15:22.0324619Z * [new branch] gh/ezyang/3171/head -> origin/gh/ezyang/3171/head 2025-11-03T16:15:22.0325095Z * [new branch] gh/ezyang/3171/orig -> origin/gh/ezyang/3171/orig 2025-11-03T16:15:22.0325234Z * [new branch] gh/ezyang/3172/base -> origin/gh/ezyang/3172/base 2025-11-03T16:15:22.0330546Z * [new branch] gh/ezyang/3172/head -> origin/gh/ezyang/3172/head 2025-11-03T16:15:22.0330857Z * [new branch] gh/ezyang/3172/orig -> origin/gh/ezyang/3172/orig 2025-11-03T16:15:22.0331015Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-11-03T16:15:22.0331220Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-11-03T16:15:22.0331365Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-11-03T16:15:22.0331603Z * [new branch] gh/ezyang/3174/base -> origin/gh/ezyang/3174/base 2025-11-03T16:15:22.0331747Z * [new branch] gh/ezyang/3174/head -> origin/gh/ezyang/3174/head 2025-11-03T16:15:22.0332343Z * [new branch] gh/ezyang/3174/orig -> origin/gh/ezyang/3174/orig 2025-11-03T16:15:22.0332664Z * [new branch] gh/ezyang/3175/base -> origin/gh/ezyang/3175/base 2025-11-03T16:15:22.0332878Z * [new branch] gh/ezyang/3175/head -> origin/gh/ezyang/3175/head 2025-11-03T16:15:22.0333025Z * [new branch] gh/ezyang/3175/orig -> origin/gh/ezyang/3175/orig 2025-11-03T16:15:22.0333238Z * [new branch] gh/ezyang/3176/base -> origin/gh/ezyang/3176/base 2025-11-03T16:15:22.0333839Z * [new branch] gh/ezyang/3176/head -> origin/gh/ezyang/3176/head 2025-11-03T16:15:22.0334003Z * [new branch] gh/ezyang/3176/orig -> origin/gh/ezyang/3176/orig 2025-11-03T16:15:22.0334317Z * [new branch] gh/ezyang/3177/base -> origin/gh/ezyang/3177/base 2025-11-03T16:15:22.0334452Z * [new branch] gh/ezyang/3177/head -> origin/gh/ezyang/3177/head 2025-11-03T16:15:22.0334583Z * [new branch] gh/ezyang/3177/orig -> origin/gh/ezyang/3177/orig 2025-11-03T16:15:22.0334716Z * [new branch] gh/ezyang/3178/base -> origin/gh/ezyang/3178/base 2025-11-03T16:15:22.0334847Z * [new branch] gh/ezyang/3178/head -> origin/gh/ezyang/3178/head 2025-11-03T16:15:22.0334979Z * [new branch] gh/ezyang/3178/orig -> origin/gh/ezyang/3178/orig 2025-11-03T16:15:22.0335103Z * [new branch] gh/ezyang/3179/base -> origin/gh/ezyang/3179/base 2025-11-03T16:15:22.0335226Z * [new branch] gh/ezyang/3179/head -> origin/gh/ezyang/3179/head 2025-11-03T16:15:22.0335364Z * [new branch] gh/ezyang/3179/orig -> origin/gh/ezyang/3179/orig 2025-11-03T16:15:22.0335487Z * [new branch] gh/ezyang/3180/base -> origin/gh/ezyang/3180/base 2025-11-03T16:15:22.0335625Z * [new branch] gh/ezyang/3180/head -> origin/gh/ezyang/3180/head 2025-11-03T16:15:22.0335752Z * [new branch] gh/ezyang/3180/orig -> origin/gh/ezyang/3180/orig 2025-11-03T16:15:22.0335895Z * [new branch] gh/ezyang/3181/base -> origin/gh/ezyang/3181/base 2025-11-03T16:15:22.0336025Z * [new branch] gh/ezyang/3181/head -> origin/gh/ezyang/3181/head 2025-11-03T16:15:22.0341551Z * [new branch] gh/ezyang/3181/orig -> origin/gh/ezyang/3181/orig 2025-11-03T16:15:22.0341859Z * [new branch] gh/ezyang/3182/base -> origin/gh/ezyang/3182/base 2025-11-03T16:15:22.0342071Z * [new branch] gh/ezyang/3182/head -> origin/gh/ezyang/3182/head 2025-11-03T16:15:22.0342239Z * [new branch] gh/ezyang/3182/orig -> origin/gh/ezyang/3182/orig 2025-11-03T16:15:22.0342363Z * [new branch] gh/ezyang/3183/base -> origin/gh/ezyang/3183/base 2025-11-03T16:15:22.0342620Z * [new branch] gh/ezyang/3183/head -> origin/gh/ezyang/3183/head 2025-11-03T16:15:22.0343354Z * [new branch] gh/ezyang/3183/orig -> origin/gh/ezyang/3183/orig 2025-11-03T16:15:22.0345231Z * [new branch] gh/ezyang/3184/base -> origin/gh/ezyang/3184/base 2025-11-03T16:15:22.0345544Z * [new branch] gh/ezyang/3184/head -> origin/gh/ezyang/3184/head 2025-11-03T16:15:22.0345757Z * [new branch] gh/ezyang/3184/orig -> origin/gh/ezyang/3184/orig 2025-11-03T16:15:22.0345970Z * [new branch] gh/ezyang/3185/base -> origin/gh/ezyang/3185/base 2025-11-03T16:15:22.0346115Z * [new branch] gh/ezyang/3185/head -> origin/gh/ezyang/3185/head 2025-11-03T16:15:22.0346256Z * [new branch] gh/ezyang/3185/orig -> origin/gh/ezyang/3185/orig 2025-11-03T16:15:22.0346471Z * [new branch] gh/ezyang/3186/base -> origin/gh/ezyang/3186/base 2025-11-03T16:15:22.0348924Z * [new branch] gh/ezyang/3186/head -> origin/gh/ezyang/3186/head 2025-11-03T16:15:22.0349171Z * [new branch] gh/ezyang/3186/orig -> origin/gh/ezyang/3186/orig 2025-11-03T16:15:22.0349310Z * [new branch] gh/ezyang/3187/base -> origin/gh/ezyang/3187/base 2025-11-03T16:15:22.0349529Z * [new branch] gh/ezyang/3187/head -> origin/gh/ezyang/3187/head 2025-11-03T16:15:22.0349669Z * [new branch] gh/ezyang/3187/orig -> origin/gh/ezyang/3187/orig 2025-11-03T16:15:22.0349873Z * [new branch] gh/ezyang/3188/base -> origin/gh/ezyang/3188/base 2025-11-03T16:15:22.0350009Z * [new branch] gh/ezyang/3188/head -> origin/gh/ezyang/3188/head 2025-11-03T16:15:22.0350341Z * [new branch] gh/ezyang/3188/orig -> origin/gh/ezyang/3188/orig 2025-11-03T16:15:22.0351380Z * [new branch] gh/ezyang/3189/base -> origin/gh/ezyang/3189/base 2025-11-03T16:15:22.0351700Z * [new branch] gh/ezyang/3189/head -> origin/gh/ezyang/3189/head 2025-11-03T16:15:22.0354743Z * [new branch] gh/ezyang/3189/orig -> origin/gh/ezyang/3189/orig 2025-11-03T16:15:22.0355073Z * [new branch] gh/ezyang/3190/base -> origin/gh/ezyang/3190/base 2025-11-03T16:15:22.0355307Z * [new branch] gh/ezyang/3190/head -> origin/gh/ezyang/3190/head 2025-11-03T16:15:22.0355451Z * [new branch] gh/ezyang/3190/orig -> origin/gh/ezyang/3190/orig 2025-11-03T16:15:22.0355609Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-11-03T16:15:22.0357481Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-11-03T16:15:22.0357829Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-11-03T16:15:22.0358062Z * [new branch] gh/fadara01/2/base -> origin/gh/fadara01/2/base 2025-11-03T16:15:22.0358315Z * [new branch] gh/fadara01/2/head -> origin/gh/fadara01/2/head 2025-11-03T16:15:22.0360630Z * [new branch] gh/fadara01/2/orig -> origin/gh/fadara01/2/orig 2025-11-03T16:15:22.0365842Z * [new branch] gh/fadara01/3/base -> origin/gh/fadara01/3/base 2025-11-03T16:15:22.0366171Z * [new branch] gh/fadara01/3/head -> origin/gh/fadara01/3/head 2025-11-03T16:15:22.0366389Z * [new branch] gh/fadara01/3/orig -> origin/gh/fadara01/3/orig 2025-11-03T16:15:22.0366537Z * [new branch] gh/fadara01/4/base -> origin/gh/fadara01/4/base 2025-11-03T16:15:22.0366660Z * [new branch] gh/fadara01/4/head -> origin/gh/fadara01/4/head 2025-11-03T16:15:22.0367174Z * [new branch] gh/fadara01/4/orig -> origin/gh/fadara01/4/orig 2025-11-03T16:15:22.0367302Z * [new branch] gh/fadara01/5/base -> origin/gh/fadara01/5/base 2025-11-03T16:15:22.0367754Z * [new branch] gh/fadara01/5/head -> origin/gh/fadara01/5/head 2025-11-03T16:15:22.0367892Z * [new branch] gh/fadara01/5/orig -> origin/gh/fadara01/5/orig 2025-11-03T16:15:22.0368020Z * [new branch] gh/fadara01/6/base -> origin/gh/fadara01/6/base 2025-11-03T16:15:22.0368161Z * [new branch] gh/fadara01/6/head -> origin/gh/fadara01/6/head 2025-11-03T16:15:22.0368286Z * [new branch] gh/fadara01/6/orig -> origin/gh/fadara01/6/orig 2025-11-03T16:15:22.0373792Z * [new branch] gh/fadara01/7/base -> origin/gh/fadara01/7/base 2025-11-03T16:15:22.0374161Z * [new branch] gh/fadara01/7/head -> origin/gh/fadara01/7/head 2025-11-03T16:15:22.0374410Z * [new branch] gh/fadara01/7/orig -> origin/gh/fadara01/7/orig 2025-11-03T16:15:22.0374555Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-11-03T16:15:22.0374823Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-11-03T16:15:22.0375460Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-11-03T16:15:22.0380395Z * [new branch] gh/fduwjj/185/base -> origin/gh/fduwjj/185/base 2025-11-03T16:15:22.0380707Z * [new branch] gh/fduwjj/185/head -> origin/gh/fduwjj/185/head 2025-11-03T16:15:22.0380932Z * [new branch] gh/fduwjj/185/orig -> origin/gh/fduwjj/185/orig 2025-11-03T16:15:22.0381149Z * [new branch] gh/fduwjj/206/base -> origin/gh/fduwjj/206/base 2025-11-03T16:15:22.0381451Z * [new branch] gh/fduwjj/206/head -> origin/gh/fduwjj/206/head 2025-11-03T16:15:22.0381589Z * [new branch] gh/fduwjj/206/orig -> origin/gh/fduwjj/206/orig 2025-11-03T16:15:22.0381861Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-11-03T16:15:22.0382496Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-11-03T16:15:22.0382671Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-11-03T16:15:22.0382806Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-11-03T16:15:22.0382937Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-11-03T16:15:22.0383071Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-11-03T16:15:22.0386978Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-11-03T16:15:22.0387321Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-11-03T16:15:22.0387553Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-11-03T16:15:22.0387710Z * [new branch] gh/fduwjj/217/base -> origin/gh/fduwjj/217/base 2025-11-03T16:15:22.0387848Z * [new branch] gh/fduwjj/217/head -> origin/gh/fduwjj/217/head 2025-11-03T16:15:22.0388111Z * [new branch] gh/fduwjj/217/orig -> origin/gh/fduwjj/217/orig 2025-11-03T16:15:22.0388901Z * [new branch] gh/fduwjj/219/base -> origin/gh/fduwjj/219/base 2025-11-03T16:15:22.0389057Z * [new branch] gh/fduwjj/219/head -> origin/gh/fduwjj/219/head 2025-11-03T16:15:22.0389185Z * [new branch] gh/fduwjj/219/orig -> origin/gh/fduwjj/219/orig 2025-11-03T16:15:22.0389426Z * [new branch] gh/fduwjj/220/base -> origin/gh/fduwjj/220/base 2025-11-03T16:15:22.0389574Z * [new branch] gh/fduwjj/220/head -> origin/gh/fduwjj/220/head 2025-11-03T16:15:22.0389785Z * [new branch] gh/fduwjj/220/orig -> origin/gh/fduwjj/220/orig 2025-11-03T16:15:22.0389967Z * [new branch] gh/fduwjj/221/base -> origin/gh/fduwjj/221/base 2025-11-03T16:15:22.0392220Z * [new branch] gh/fduwjj/221/head -> origin/gh/fduwjj/221/head 2025-11-03T16:15:22.0392392Z * [new branch] gh/fduwjj/221/orig -> origin/gh/fduwjj/221/orig 2025-11-03T16:15:22.0392519Z * [new branch] gh/fduwjj/222/base -> origin/gh/fduwjj/222/base 2025-11-03T16:15:22.0392697Z * [new branch] gh/fduwjj/222/head -> origin/gh/fduwjj/222/head 2025-11-03T16:15:22.0394361Z * [new branch] gh/fduwjj/222/orig -> origin/gh/fduwjj/222/orig 2025-11-03T16:15:22.0394519Z * [new branch] gh/fduwjj/223/base -> origin/gh/fduwjj/223/base 2025-11-03T16:15:22.0394949Z * [new branch] gh/fduwjj/223/head -> origin/gh/fduwjj/223/head 2025-11-03T16:15:22.0396959Z * [new branch] gh/fduwjj/223/orig -> origin/gh/fduwjj/223/orig 2025-11-03T16:15:22.0397271Z * [new branch] gh/fduwjj/224/base -> origin/gh/fduwjj/224/base 2025-11-03T16:15:22.0397519Z * [new branch] gh/fduwjj/224/head -> origin/gh/fduwjj/224/head 2025-11-03T16:15:22.0397665Z * [new branch] gh/fduwjj/224/orig -> origin/gh/fduwjj/224/orig 2025-11-03T16:15:22.0399814Z * [new branch] gh/fduwjj/225/base -> origin/gh/fduwjj/225/base 2025-11-03T16:15:22.0400136Z * [new branch] gh/fduwjj/225/head -> origin/gh/fduwjj/225/head 2025-11-03T16:15:22.0400357Z * [new branch] gh/fduwjj/225/orig -> origin/gh/fduwjj/225/orig 2025-11-03T16:15:22.0400628Z * [new branch] gh/fduwjj/226/base -> origin/gh/fduwjj/226/base 2025-11-03T16:15:22.0401743Z * [new branch] gh/fduwjj/226/head -> origin/gh/fduwjj/226/head 2025-11-03T16:15:22.0402015Z * [new branch] gh/fduwjj/226/orig -> origin/gh/fduwjj/226/orig 2025-11-03T16:15:22.0404446Z * [new branch] gh/fduwjj/227/base -> origin/gh/fduwjj/227/base 2025-11-03T16:15:22.0404768Z * [new branch] gh/fduwjj/227/head -> origin/gh/fduwjj/227/head 2025-11-03T16:15:22.0404987Z * [new branch] gh/fduwjj/227/orig -> origin/gh/fduwjj/227/orig 2025-11-03T16:15:22.0405127Z * [new branch] gh/fduwjj/228/base -> origin/gh/fduwjj/228/base 2025-11-03T16:15:22.0406220Z * [new branch] gh/fduwjj/228/head -> origin/gh/fduwjj/228/head 2025-11-03T16:15:22.0406650Z * [new branch] gh/fduwjj/228/orig -> origin/gh/fduwjj/228/orig 2025-11-03T16:15:22.0408413Z * [new branch] gh/fduwjj/229/base -> origin/gh/fduwjj/229/base 2025-11-03T16:15:22.0408737Z * [new branch] gh/fduwjj/229/head -> origin/gh/fduwjj/229/head 2025-11-03T16:15:22.0408954Z * [new branch] gh/fduwjj/229/orig -> origin/gh/fduwjj/229/orig 2025-11-03T16:15:22.0410770Z * [new branch] gh/fduwjj/230/base -> origin/gh/fduwjj/230/base 2025-11-03T16:15:22.0411087Z * [new branch] gh/fduwjj/230/head -> origin/gh/fduwjj/230/head 2025-11-03T16:15:22.0411292Z * [new branch] gh/fduwjj/230/orig -> origin/gh/fduwjj/230/orig 2025-11-03T16:15:22.0413009Z * [new branch] gh/fduwjj/231/base -> origin/gh/fduwjj/231/base 2025-11-03T16:15:22.0413465Z * [new branch] gh/fduwjj/231/head -> origin/gh/fduwjj/231/head 2025-11-03T16:15:22.0413613Z * [new branch] gh/fduwjj/231/orig -> origin/gh/fduwjj/231/orig 2025-11-03T16:15:22.0414084Z * [new branch] gh/fduwjj/232/base -> origin/gh/fduwjj/232/base 2025-11-03T16:15:22.0415322Z * [new branch] gh/fduwjj/232/head -> origin/gh/fduwjj/232/head 2025-11-03T16:15:22.0415634Z * [new branch] gh/fduwjj/232/orig -> origin/gh/fduwjj/232/orig 2025-11-03T16:15:22.0416134Z * [new branch] gh/fduwjj/233/base -> origin/gh/fduwjj/233/base 2025-11-03T16:15:22.0418023Z * [new branch] gh/fduwjj/233/head -> origin/gh/fduwjj/233/head 2025-11-03T16:15:22.0418335Z * [new branch] gh/fduwjj/233/orig -> origin/gh/fduwjj/233/orig 2025-11-03T16:15:22.0418558Z * [new branch] gh/fegin/318/base -> origin/gh/fegin/318/base 2025-11-03T16:15:22.0418913Z * [new branch] gh/fegin/318/head -> origin/gh/fegin/318/head 2025-11-03T16:15:22.0423074Z * [new branch] gh/fegin/318/orig -> origin/gh/fegin/318/orig 2025-11-03T16:15:22.0423385Z * [new branch] gh/fegin/321/base -> origin/gh/fegin/321/base 2025-11-03T16:15:22.0423612Z * [new branch] gh/fegin/321/head -> origin/gh/fegin/321/head 2025-11-03T16:15:22.0423746Z * [new branch] gh/fegin/321/orig -> origin/gh/fegin/321/orig 2025-11-03T16:15:22.0423967Z * [new branch] gh/fegin/325/base -> origin/gh/fegin/325/base 2025-11-03T16:15:22.0424097Z * [new branch] gh/fegin/325/head -> origin/gh/fegin/325/head 2025-11-03T16:15:22.0424735Z * [new branch] gh/fegin/325/orig -> origin/gh/fegin/325/orig 2025-11-03T16:15:22.0425050Z * [new branch] gh/fegin/326/base -> origin/gh/fegin/326/base 2025-11-03T16:15:22.0425330Z * [new branch] gh/fegin/326/head -> origin/gh/fegin/326/head 2025-11-03T16:15:22.0426517Z * [new branch] gh/fegin/326/orig -> origin/gh/fegin/326/orig 2025-11-03T16:15:22.0426771Z * [new branch] gh/fegin/327/base -> origin/gh/fegin/327/base 2025-11-03T16:15:22.0429487Z * [new branch] gh/fegin/327/head -> origin/gh/fegin/327/head 2025-11-03T16:15:22.0429826Z * [new branch] gh/fegin/327/orig -> origin/gh/fegin/327/orig 2025-11-03T16:15:22.0430059Z * [new branch] gh/fegin/328/base -> origin/gh/fegin/328/base 2025-11-03T16:15:22.0430191Z * [new branch] gh/fegin/328/head -> origin/gh/fegin/328/head 2025-11-03T16:15:22.0430388Z * [new branch] gh/fegin/328/orig -> origin/gh/fegin/328/orig 2025-11-03T16:15:22.0431271Z * [new branch] gh/fegin/329/base -> origin/gh/fegin/329/base 2025-11-03T16:15:22.0431742Z * [new branch] gh/fegin/329/head -> origin/gh/fegin/329/head 2025-11-03T16:15:22.0432368Z * [new branch] gh/fegin/329/orig -> origin/gh/fegin/329/orig 2025-11-03T16:15:22.0437079Z * [new branch] gh/fegin/330/base -> origin/gh/fegin/330/base 2025-11-03T16:15:22.0437387Z * [new branch] gh/fegin/330/head -> origin/gh/fegin/330/head 2025-11-03T16:15:22.0437592Z * [new branch] gh/fegin/330/orig -> origin/gh/fegin/330/orig 2025-11-03T16:15:22.0437734Z * [new branch] gh/fegin/331/base -> origin/gh/fegin/331/base 2025-11-03T16:15:22.0437941Z * [new branch] gh/fegin/331/head -> origin/gh/fegin/331/head 2025-11-03T16:15:22.0438073Z * [new branch] gh/fegin/331/orig -> origin/gh/fegin/331/orig 2025-11-03T16:15:22.0438285Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-11-03T16:15:22.0439466Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-11-03T16:15:22.0439775Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-11-03T16:15:22.0440086Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-11-03T16:15:22.0442200Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-11-03T16:15:22.0442519Z * [new branch] gh/fffrog/171/base -> origin/gh/fffrog/171/base 2025-11-03T16:15:22.0442906Z * [new branch] gh/fffrog/171/head -> origin/gh/fffrog/171/head 2025-11-03T16:15:22.0443052Z * [new branch] gh/fffrog/171/orig -> origin/gh/fffrog/171/orig 2025-11-03T16:15:22.0443900Z * [new branch] gh/fffrog/175/base -> origin/gh/fffrog/175/base 2025-11-03T16:15:22.0444230Z * [new branch] gh/fffrog/175/head -> origin/gh/fffrog/175/head 2025-11-03T16:15:22.0446419Z * [new branch] gh/fffrog/175/orig -> origin/gh/fffrog/175/orig 2025-11-03T16:15:22.0446747Z * [new branch] gh/fffrog/176/base -> origin/gh/fffrog/176/base 2025-11-03T16:15:22.0446958Z * [new branch] gh/fffrog/176/head -> origin/gh/fffrog/176/head 2025-11-03T16:15:22.0447123Z * [new branch] gh/fffrog/176/orig -> origin/gh/fffrog/176/orig 2025-11-03T16:15:22.0451034Z * [new branch] gh/fffrog/177/base -> origin/gh/fffrog/177/base 2025-11-03T16:15:22.0451376Z * [new branch] gh/fffrog/177/head -> origin/gh/fffrog/177/head 2025-11-03T16:15:22.0451588Z * [new branch] gh/fffrog/177/orig -> origin/gh/fffrog/177/orig 2025-11-03T16:15:22.0451804Z * [new branch] gh/fffrog/178/base -> origin/gh/fffrog/178/base 2025-11-03T16:15:22.0451941Z * [new branch] gh/fffrog/178/head -> origin/gh/fffrog/178/head 2025-11-03T16:15:22.0452152Z * [new branch] gh/fffrog/178/orig -> origin/gh/fffrog/178/orig 2025-11-03T16:15:22.0452786Z * [new branch] gh/fffrog/179/base -> origin/gh/fffrog/179/base 2025-11-03T16:15:22.0453104Z * [new branch] gh/fffrog/179/head -> origin/gh/fffrog/179/head 2025-11-03T16:15:22.0453341Z * [new branch] gh/fffrog/179/orig -> origin/gh/fffrog/179/orig 2025-11-03T16:15:22.0458007Z * [new branch] gh/fffrog/180/base -> origin/gh/fffrog/180/base 2025-11-03T16:15:22.0458354Z * [new branch] gh/fffrog/180/head -> origin/gh/fffrog/180/head 2025-11-03T16:15:22.0458586Z * [new branch] gh/fffrog/180/orig -> origin/gh/fffrog/180/orig 2025-11-03T16:15:22.0458735Z * [new branch] gh/fffrog/181/base -> origin/gh/fffrog/181/base 2025-11-03T16:15:22.0458969Z * [new branch] gh/fffrog/181/head -> origin/gh/fffrog/181/head 2025-11-03T16:15:22.0459117Z * [new branch] gh/fffrog/181/orig -> origin/gh/fffrog/181/orig 2025-11-03T16:15:22.0459751Z * [new branch] gh/fffrog/182/base -> origin/gh/fffrog/182/base 2025-11-03T16:15:22.0459941Z * [new branch] gh/fffrog/182/head -> origin/gh/fffrog/182/head 2025-11-03T16:15:22.0460076Z * [new branch] gh/fffrog/182/orig -> origin/gh/fffrog/182/orig 2025-11-03T16:15:22.0461496Z * [new branch] gh/fxdawnn/1/base -> origin/gh/fxdawnn/1/base 2025-11-03T16:15:22.0461847Z * [new branch] gh/fxdawnn/1/head -> origin/gh/fxdawnn/1/head 2025-11-03T16:15:22.0462067Z * [new branch] gh/fxdawnn/1/orig -> origin/gh/fxdawnn/1/orig 2025-11-03T16:15:22.0464539Z * [new branch] gh/fxdawnn/2/base -> origin/gh/fxdawnn/2/base 2025-11-03T16:15:22.0464846Z * [new branch] gh/fxdawnn/2/head -> origin/gh/fxdawnn/2/head 2025-11-03T16:15:22.0465082Z * [new branch] gh/fxdawnn/2/orig -> origin/gh/fxdawnn/2/orig 2025-11-03T16:15:22.0466446Z * [new branch] gh/fxdawnn/3/base -> origin/gh/fxdawnn/3/base 2025-11-03T16:15:22.0466785Z * [new branch] gh/fxdawnn/3/head -> origin/gh/fxdawnn/3/head 2025-11-03T16:15:22.0467013Z * [new branch] gh/fxdawnn/3/orig -> origin/gh/fxdawnn/3/orig 2025-11-03T16:15:22.0467161Z * [new branch] gh/fxdawnn/4/base -> origin/gh/fxdawnn/4/base 2025-11-03T16:15:22.0467649Z * [new branch] gh/fxdawnn/4/orig -> origin/gh/fxdawnn/4/orig 2025-11-03T16:15:22.0469106Z * [new branch] gh/fxdawnn/5/base -> origin/gh/fxdawnn/5/base 2025-11-03T16:15:22.0469280Z * [new branch] gh/fxdawnn/5/head -> origin/gh/fxdawnn/5/head 2025-11-03T16:15:22.0472259Z * [new branch] gh/fxdawnn/5/orig -> origin/gh/fxdawnn/5/orig 2025-11-03T16:15:22.0472595Z * [new branch] gh/fxdawnn/6/base -> origin/gh/fxdawnn/6/base 2025-11-03T16:15:22.0472734Z * [new branch] gh/fxdawnn/6/head -> origin/gh/fxdawnn/6/head 2025-11-03T16:15:22.0472879Z * [new branch] gh/fxdawnn/6/orig -> origin/gh/fxdawnn/6/orig 2025-11-03T16:15:22.0474986Z * [new branch] gh/fxdawnn/7/base -> origin/gh/fxdawnn/7/base 2025-11-03T16:15:22.0475311Z * [new branch] gh/fxdawnn/7/head -> origin/gh/fxdawnn/7/head 2025-11-03T16:15:22.0475562Z * [new branch] gh/fxdawnn/7/orig -> origin/gh/fxdawnn/7/orig 2025-11-03T16:15:22.0475730Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-11-03T16:15:22.0475973Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-11-03T16:15:22.0476121Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-11-03T16:15:22.0478007Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-11-03T16:15:22.0478196Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-11-03T16:15:22.0480353Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-11-03T16:15:22.0480707Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-11-03T16:15:22.0480954Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-11-03T16:15:22.0481106Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-11-03T16:15:22.0485327Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-11-03T16:15:22.0485637Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-11-03T16:15:22.0485878Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-11-03T16:15:22.0486030Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-11-03T16:15:22.0486162Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-11-03T16:15:22.0486422Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-11-03T16:15:22.0486810Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-11-03T16:15:22.0487641Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-11-03T16:15:22.0488054Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-11-03T16:15:22.0490674Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-11-03T16:15:22.0490993Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-11-03T16:15:22.0491226Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-11-03T16:15:22.0491378Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-11-03T16:15:22.0491762Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-11-03T16:15:22.0492579Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-11-03T16:15:22.0493910Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-11-03T16:15:22.0494210Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-11-03T16:15:22.0494487Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-11-03T16:15:22.0496283Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-11-03T16:15:22.0496602Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-11-03T16:15:22.0496821Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-11-03T16:15:22.0498637Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-11-03T16:15:22.0498965Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-11-03T16:15:22.0499183Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-11-03T16:15:22.0499325Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-11-03T16:15:22.0500771Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-11-03T16:15:22.0501070Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-11-03T16:15:22.0501529Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-11-03T16:15:22.0502637Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-11-03T16:15:22.0503126Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-11-03T16:15:22.0503818Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-11-03T16:15:22.0504347Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-11-03T16:15:22.0506307Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-11-03T16:15:22.0506621Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-11-03T16:15:22.0506852Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-11-03T16:15:22.0506998Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-11-03T16:15:22.0508237Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-11-03T16:15:22.0508401Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-11-03T16:15:22.0508936Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-11-03T16:15:22.0512605Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-11-03T16:15:22.0512783Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-11-03T16:15:22.0512927Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-11-03T16:15:22.0513093Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-11-03T16:15:22.0513382Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-11-03T16:15:22.0513529Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-11-03T16:15:22.0520789Z * [new branch] gh/guangyey/195/base -> origin/gh/guangyey/195/base 2025-11-03T16:15:22.0525552Z * [new branch] gh/guangyey/195/head -> origin/gh/guangyey/195/head 2025-11-03T16:15:22.0527649Z * [new branch] gh/guangyey/195/orig -> origin/gh/guangyey/195/orig 2025-11-03T16:15:22.0527814Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-11-03T16:15:22.0527959Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-11-03T16:15:22.0528192Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-11-03T16:15:22.0532775Z * [new branch] gh/guangyey/210/base -> origin/gh/guangyey/210/base 2025-11-03T16:15:22.0533102Z * [new branch] gh/guangyey/210/head -> origin/gh/guangyey/210/head 2025-11-03T16:15:22.0533312Z * [new branch] gh/guangyey/210/orig -> origin/gh/guangyey/210/orig 2025-11-03T16:15:22.0533469Z * [new branch] gh/guangyey/212/base -> origin/gh/guangyey/212/base 2025-11-03T16:15:22.0533700Z * [new branch] gh/guangyey/212/head -> origin/gh/guangyey/212/head 2025-11-03T16:15:22.0533845Z * [new branch] gh/guangyey/212/orig -> origin/gh/guangyey/212/orig 2025-11-03T16:15:22.0534063Z * [new branch] gh/guangyey/213/base -> origin/gh/guangyey/213/base 2025-11-03T16:15:22.0534207Z * [new branch] gh/guangyey/213/head -> origin/gh/guangyey/213/head 2025-11-03T16:15:22.0534411Z * [new branch] gh/guangyey/213/orig -> origin/gh/guangyey/213/orig 2025-11-03T16:15:22.0534540Z * [new branch] gh/guangyey/214/base -> origin/gh/guangyey/214/base 2025-11-03T16:15:22.0534673Z * [new branch] gh/guangyey/214/head -> origin/gh/guangyey/214/head 2025-11-03T16:15:22.0535102Z * [new branch] gh/guangyey/214/orig -> origin/gh/guangyey/214/orig 2025-11-03T16:15:22.0535252Z * [new branch] gh/guangyey/215/base -> origin/gh/guangyey/215/base 2025-11-03T16:15:22.0535382Z * [new branch] gh/guangyey/215/head -> origin/gh/guangyey/215/head 2025-11-03T16:15:22.0535505Z * [new branch] gh/guangyey/215/orig -> origin/gh/guangyey/215/orig 2025-11-03T16:15:22.0535820Z * [new branch] gh/guangyey/216/base -> origin/gh/guangyey/216/base 2025-11-03T16:15:22.0536117Z * [new branch] gh/guangyey/216/head -> origin/gh/guangyey/216/head 2025-11-03T16:15:22.0536672Z * [new branch] gh/guangyey/216/orig -> origin/gh/guangyey/216/orig 2025-11-03T16:15:22.0537813Z * [new branch] gh/guangyey/217/base -> origin/gh/guangyey/217/base 2025-11-03T16:15:22.0538077Z * [new branch] gh/guangyey/217/head -> origin/gh/guangyey/217/head 2025-11-03T16:15:22.0538536Z * [new branch] gh/guangyey/217/orig -> origin/gh/guangyey/217/orig 2025-11-03T16:15:22.0542603Z * [new branch] gh/guangyey/218/base -> origin/gh/guangyey/218/base 2025-11-03T16:15:22.0542921Z * [new branch] gh/guangyey/218/head -> origin/gh/guangyey/218/head 2025-11-03T16:15:22.0543145Z * [new branch] gh/guangyey/218/orig -> origin/gh/guangyey/218/orig 2025-11-03T16:15:22.0543304Z * [new branch] gh/guangyey/219/base -> origin/gh/guangyey/219/base 2025-11-03T16:15:22.0543546Z * [new branch] gh/guangyey/219/head -> origin/gh/guangyey/219/head 2025-11-03T16:15:22.0543695Z * [new branch] gh/guangyey/219/orig -> origin/gh/guangyey/219/orig 2025-11-03T16:15:22.0543969Z * [new branch] gh/guangyey/220/base -> origin/gh/guangyey/220/base 2025-11-03T16:15:22.0545327Z * [new branch] gh/guangyey/220/head -> origin/gh/guangyey/220/head 2025-11-03T16:15:22.0545598Z * [new branch] gh/guangyey/220/orig -> origin/gh/guangyey/220/orig 2025-11-03T16:15:22.0547877Z * [new branch] gh/guangyey/221/base -> origin/gh/guangyey/221/base 2025-11-03T16:15:22.0548210Z * [new branch] gh/guangyey/221/head -> origin/gh/guangyey/221/head 2025-11-03T16:15:22.0548430Z * [new branch] gh/guangyey/221/orig -> origin/gh/guangyey/221/orig 2025-11-03T16:15:22.0548592Z * [new branch] gh/guangyey/222/base -> origin/gh/guangyey/222/base 2025-11-03T16:15:22.0548894Z * [new branch] gh/guangyey/222/head -> origin/gh/guangyey/222/head 2025-11-03T16:15:22.0550176Z * [new branch] gh/guangyey/222/orig -> origin/gh/guangyey/222/orig 2025-11-03T16:15:22.0550722Z * [new branch] gh/guangyey/223/base -> origin/gh/guangyey/223/base 2025-11-03T16:15:22.0551064Z * [new branch] gh/guangyey/223/head -> origin/gh/guangyey/223/head 2025-11-03T16:15:22.0551741Z * [new branch] gh/guangyey/223/orig -> origin/gh/guangyey/223/orig 2025-11-03T16:15:22.0552879Z * [new branch] gh/guangyey/224/base -> origin/gh/guangyey/224/base 2025-11-03T16:15:22.0553232Z * [new branch] gh/guangyey/224/head -> origin/gh/guangyey/224/head 2025-11-03T16:15:22.0553965Z * [new branch] gh/guangyey/224/orig -> origin/gh/guangyey/224/orig 2025-11-03T16:15:22.0555530Z * [new branch] gh/guangyey/225/base -> origin/gh/guangyey/225/base 2025-11-03T16:15:22.0556059Z * [new branch] gh/guangyey/225/head -> origin/gh/guangyey/225/head 2025-11-03T16:15:22.0559127Z * [new branch] gh/guangyey/225/orig -> origin/gh/guangyey/225/orig 2025-11-03T16:15:22.0564636Z * [new branch] gh/guangyey/226/base -> origin/gh/guangyey/226/base 2025-11-03T16:15:22.0566502Z * [new branch] gh/guangyey/226/head -> origin/gh/guangyey/226/head 2025-11-03T16:15:22.0566998Z * [new branch] gh/guangyey/226/orig -> origin/gh/guangyey/226/orig 2025-11-03T16:15:22.0571417Z * [new branch] gh/guangyey/227/base -> origin/gh/guangyey/227/base 2025-11-03T16:15:22.0575792Z * [new branch] gh/guangyey/227/head -> origin/gh/guangyey/227/head 2025-11-03T16:15:22.0578100Z * [new branch] gh/guangyey/227/orig -> origin/gh/guangyey/227/orig 2025-11-03T16:15:22.0578677Z * [new branch] gh/guangyey/228/base -> origin/gh/guangyey/228/base 2025-11-03T16:15:22.0579129Z * [new branch] gh/guangyey/228/head -> origin/gh/guangyey/228/head 2025-11-03T16:15:22.0579595Z * [new branch] gh/guangyey/228/orig -> origin/gh/guangyey/228/orig 2025-11-03T16:15:22.0579936Z * [new branch] gh/guangyey/229/base -> origin/gh/guangyey/229/base 2025-11-03T16:15:22.0580283Z * [new branch] gh/guangyey/229/head -> origin/gh/guangyey/229/head 2025-11-03T16:15:22.0580608Z * [new branch] gh/guangyey/229/orig -> origin/gh/guangyey/229/orig 2025-11-03T16:15:22.0580938Z * [new branch] gh/guangyey/230/base -> origin/gh/guangyey/230/base 2025-11-03T16:15:22.0581252Z * [new branch] gh/guangyey/230/head -> origin/gh/guangyey/230/head 2025-11-03T16:15:22.0581607Z * [new branch] gh/guangyey/230/orig -> origin/gh/guangyey/230/orig 2025-11-03T16:15:22.0581977Z * [new branch] gh/guangyey/231/base -> origin/gh/guangyey/231/base 2025-11-03T16:15:22.0582321Z * [new branch] gh/guangyey/231/head -> origin/gh/guangyey/231/head 2025-11-03T16:15:22.0582645Z * [new branch] gh/guangyey/231/orig -> origin/gh/guangyey/231/orig 2025-11-03T16:15:22.0582962Z * [new branch] gh/guangyey/232/base -> origin/gh/guangyey/232/base 2025-11-03T16:15:22.0583287Z * [new branch] gh/guangyey/232/head -> origin/gh/guangyey/232/head 2025-11-03T16:15:22.0583617Z * [new branch] gh/guangyey/232/orig -> origin/gh/guangyey/232/orig 2025-11-03T16:15:22.0583947Z * [new branch] gh/guangyey/233/base -> origin/gh/guangyey/233/base 2025-11-03T16:15:22.0584256Z * [new branch] gh/guangyey/233/head -> origin/gh/guangyey/233/head 2025-11-03T16:15:22.0584576Z * [new branch] gh/guangyey/233/orig -> origin/gh/guangyey/233/orig 2025-11-03T16:15:22.0584895Z * [new branch] gh/guangyey/234/base -> origin/gh/guangyey/234/base 2025-11-03T16:15:22.0585270Z * [new branch] gh/guangyey/234/head -> origin/gh/guangyey/234/head 2025-11-03T16:15:22.0585583Z * [new branch] gh/guangyey/234/orig -> origin/gh/guangyey/234/orig 2025-11-03T16:15:22.0585906Z * [new branch] gh/guangyey/235/base -> origin/gh/guangyey/235/base 2025-11-03T16:15:22.0586230Z * [new branch] gh/guangyey/235/head -> origin/gh/guangyey/235/head 2025-11-03T16:15:22.0586549Z * [new branch] gh/guangyey/235/orig -> origin/gh/guangyey/235/orig 2025-11-03T16:15:22.0586903Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-11-03T16:15:22.0587277Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-11-03T16:15:22.0587633Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-11-03T16:15:22.0587991Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-11-03T16:15:22.0588348Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-11-03T16:15:22.0588703Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-11-03T16:15:22.0589059Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-11-03T16:15:22.0589407Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-11-03T16:15:22.0589767Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-11-03T16:15:22.0590126Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-11-03T16:15:22.0590519Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-11-03T16:15:22.0590887Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-11-03T16:15:22.0591248Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-11-03T16:15:22.0591644Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-11-03T16:15:22.0591992Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-11-03T16:15:22.0592351Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-11-03T16:15:22.0592709Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-11-03T16:15:22.0593077Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-11-03T16:15:22.0593598Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-11-03T16:15:22.0594078Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-11-03T16:15:22.0594483Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-11-03T16:15:22.0596546Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-11-03T16:15:22.0597120Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-11-03T16:15:22.0597619Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-11-03T16:15:22.0598112Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-11-03T16:15:22.0598594Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-11-03T16:15:22.0599450Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-11-03T16:15:22.0599981Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-11-03T16:15:22.0600555Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-11-03T16:15:22.0601098Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-11-03T16:15:22.0603531Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-11-03T16:15:22.0604100Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-11-03T16:15:22.0604593Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-11-03T16:15:22.0605091Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-11-03T16:15:22.0605585Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-11-03T16:15:22.0605949Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-11-03T16:15:22.0606383Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-11-03T16:15:22.0607077Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-11-03T16:15:22.0607764Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-11-03T16:15:22.0608820Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-11-03T16:15:22.0609187Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-11-03T16:15:22.0609848Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-11-03T16:15:22.0611609Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-11-03T16:15:22.0612315Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-11-03T16:15:22.0612819Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-11-03T16:15:22.0613457Z * [new branch] gh/guilhermeleobas/239/base -> origin/gh/guilhermeleobas/239/base 2025-11-03T16:15:22.0618003Z * [new branch] gh/guilhermeleobas/239/head -> origin/gh/guilhermeleobas/239/head 2025-11-03T16:15:22.0619724Z * [new branch] gh/guilhermeleobas/239/orig -> origin/gh/guilhermeleobas/239/orig 2025-11-03T16:15:22.0624954Z * [new branch] gh/guilhermeleobas/246/base -> origin/gh/guilhermeleobas/246/base 2025-11-03T16:15:22.0626946Z * [new branch] gh/guilhermeleobas/246/head -> origin/gh/guilhermeleobas/246/head 2025-11-03T16:15:22.0627519Z * [new branch] gh/guilhermeleobas/246/orig -> origin/gh/guilhermeleobas/246/orig 2025-11-03T16:15:22.0628044Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-11-03T16:15:22.0628542Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-11-03T16:15:22.0629050Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-11-03T16:15:22.0629424Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-11-03T16:15:22.0629804Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-11-03T16:15:22.0630168Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-11-03T16:15:22.0630530Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-11-03T16:15:22.0630885Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-11-03T16:15:22.0631256Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-11-03T16:15:22.0631614Z * [new branch] gh/guilhermeleobas/251/base -> origin/gh/guilhermeleobas/251/base 2025-11-03T16:15:22.0631981Z * [new branch] gh/guilhermeleobas/251/head -> origin/gh/guilhermeleobas/251/head 2025-11-03T16:15:22.0632535Z * [new branch] gh/guilhermeleobas/251/orig -> origin/gh/guilhermeleobas/251/orig 2025-11-03T16:15:22.0632896Z * [new branch] gh/guilhermeleobas/252/base -> origin/gh/guilhermeleobas/252/base 2025-11-03T16:15:22.0633258Z * [new branch] gh/guilhermeleobas/252/head -> origin/gh/guilhermeleobas/252/head 2025-11-03T16:15:22.0633640Z * [new branch] gh/guilhermeleobas/252/orig -> origin/gh/guilhermeleobas/252/orig 2025-11-03T16:15:22.0634183Z * [new branch] gh/henrylhtsang/150/base -> origin/gh/henrylhtsang/150/base 2025-11-03T16:15:22.0634568Z * [new branch] gh/henrylhtsang/150/head -> origin/gh/henrylhtsang/150/head 2025-11-03T16:15:22.0634942Z * [new branch] gh/henrylhtsang/150/orig -> origin/gh/henrylhtsang/150/orig 2025-11-03T16:15:22.0635302Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-11-03T16:15:22.0635642Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-11-03T16:15:22.0635949Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-11-03T16:15:22.0636253Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-11-03T16:15:22.0641810Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-11-03T16:15:22.0644274Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-11-03T16:15:22.0644748Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-11-03T16:15:22.0649940Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-11-03T16:15:22.0652885Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-11-03T16:15:22.0653439Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-11-03T16:15:22.0654260Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-11-03T16:15:22.0654661Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-11-03T16:15:22.0655015Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-11-03T16:15:22.0655332Z * [new branch] gh/isuruf/157/base -> origin/gh/isuruf/157/base 2025-11-03T16:15:22.0655644Z * [new branch] gh/isuruf/157/head -> origin/gh/isuruf/157/head 2025-11-03T16:15:22.0655950Z * [new branch] gh/isuruf/157/orig -> origin/gh/isuruf/157/orig 2025-11-03T16:15:22.0656276Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-11-03T16:15:22.0656580Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-11-03T16:15:22.0656886Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-11-03T16:15:22.0657215Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-11-03T16:15:22.0657533Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-11-03T16:15:22.0657853Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-11-03T16:15:22.0658153Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-11-03T16:15:22.0658458Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-11-03T16:15:22.0658762Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-11-03T16:15:22.0659070Z * [new branch] gh/jamesjwu/195/base -> origin/gh/jamesjwu/195/base 2025-11-03T16:15:22.0659376Z * [new branch] gh/jamesjwu/195/head -> origin/gh/jamesjwu/195/head 2025-11-03T16:15:22.0659671Z * [new branch] gh/jamesjwu/195/orig -> origin/gh/jamesjwu/195/orig 2025-11-03T16:15:22.0660114Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-11-03T16:15:22.0660436Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-11-03T16:15:22.0660754Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-11-03T16:15:22.0661098Z * [new branch] gh/jamesjwu/197/base -> origin/gh/jamesjwu/197/base 2025-11-03T16:15:22.0661423Z * [new branch] gh/jamesjwu/197/head -> origin/gh/jamesjwu/197/head 2025-11-03T16:15:22.0661760Z * [new branch] gh/jamesjwu/197/orig -> origin/gh/jamesjwu/197/orig 2025-11-03T16:15:22.0662082Z * [new branch] gh/jamesjwu/198/base -> origin/gh/jamesjwu/198/base 2025-11-03T16:15:22.0662391Z * [new branch] gh/jamesjwu/198/head -> origin/gh/jamesjwu/198/head 2025-11-03T16:15:22.0662713Z * [new branch] gh/jamesjwu/198/orig -> origin/gh/jamesjwu/198/orig 2025-11-03T16:15:22.0663032Z * [new branch] gh/jamesjwu/199/base -> origin/gh/jamesjwu/199/base 2025-11-03T16:15:22.0663349Z * [new branch] gh/jamesjwu/199/head -> origin/gh/jamesjwu/199/head 2025-11-03T16:15:22.0663667Z * [new branch] gh/jamesjwu/199/orig -> origin/gh/jamesjwu/199/orig 2025-11-03T16:15:22.0663976Z * [new branch] gh/jamesjwu/200/base -> origin/gh/jamesjwu/200/base 2025-11-03T16:15:22.0664291Z * [new branch] gh/jamesjwu/200/head -> origin/gh/jamesjwu/200/head 2025-11-03T16:15:22.0664741Z * [new branch] gh/jamesjwu/200/orig -> origin/gh/jamesjwu/200/orig 2025-11-03T16:15:22.0665134Z * [new branch] gh/jamesjwu/201/base -> origin/gh/jamesjwu/201/base 2025-11-03T16:15:22.0665453Z * [new branch] gh/jamesjwu/201/head -> origin/gh/jamesjwu/201/head 2025-11-03T16:15:22.0665763Z * [new branch] gh/jamesjwu/201/orig -> origin/gh/jamesjwu/201/orig 2025-11-03T16:15:22.0666078Z * [new branch] gh/jamesjwu/202/base -> origin/gh/jamesjwu/202/base 2025-11-03T16:15:22.0666390Z * [new branch] gh/jamesjwu/202/head -> origin/gh/jamesjwu/202/head 2025-11-03T16:15:22.0666708Z * [new branch] gh/jamesjwu/202/orig -> origin/gh/jamesjwu/202/orig 2025-11-03T16:15:22.0667021Z * [new branch] gh/jamesjwu/203/base -> origin/gh/jamesjwu/203/base 2025-11-03T16:15:22.0667527Z * [new branch] gh/jamesjwu/203/head -> origin/gh/jamesjwu/203/head 2025-11-03T16:15:22.0667961Z * [new branch] gh/jamesjwu/203/orig -> origin/gh/jamesjwu/203/orig 2025-11-03T16:15:22.0669006Z * [new branch] gh/jamesjwu/204/base -> origin/gh/jamesjwu/204/base 2025-11-03T16:15:22.0669437Z * [new branch] gh/jamesjwu/204/head -> origin/gh/jamesjwu/204/head 2025-11-03T16:15:22.0704238Z * [new branch] gh/jamesjwu/204/orig -> origin/gh/jamesjwu/204/orig 2025-11-03T16:15:22.0704883Z * [new branch] gh/jamesjwu/205/base -> origin/gh/jamesjwu/205/base 2025-11-03T16:15:22.0705334Z * [new branch] gh/jamesjwu/205/head -> origin/gh/jamesjwu/205/head 2025-11-03T16:15:22.0705656Z * [new branch] gh/jamesjwu/205/orig -> origin/gh/jamesjwu/205/orig 2025-11-03T16:15:22.0706009Z * [new branch] gh/jamesjwu/206/base -> origin/gh/jamesjwu/206/base 2025-11-03T16:15:22.0706322Z * [new branch] gh/jamesjwu/206/head -> origin/gh/jamesjwu/206/head 2025-11-03T16:15:22.0706646Z * [new branch] gh/jamesjwu/206/orig -> origin/gh/jamesjwu/206/orig 2025-11-03T16:15:22.0706949Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-11-03T16:15:22.0707256Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-11-03T16:15:22.0707689Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-11-03T16:15:22.0707996Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-11-03T16:15:22.0708298Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-11-03T16:15:22.0708590Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-11-03T16:15:22.0708891Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-11-03T16:15:22.0709190Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-11-03T16:15:22.0709498Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-11-03T16:15:22.0709801Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-11-03T16:15:22.0710096Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-11-03T16:15:22.0710406Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-11-03T16:15:22.0710707Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-11-03T16:15:22.0711006Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-11-03T16:15:22.0711301Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-11-03T16:15:22.0711588Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-11-03T16:15:22.0711884Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-11-03T16:15:22.0712218Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-11-03T16:15:22.0712521Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-11-03T16:15:22.0712852Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-11-03T16:15:22.0713172Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-11-03T16:15:22.0713646Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-11-03T16:15:22.0714058Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-11-03T16:15:22.0714370Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-11-03T16:15:22.0714670Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-11-03T16:15:22.0714976Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-11-03T16:15:22.0715298Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-11-03T16:15:22.0715618Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-11-03T16:15:22.0715934Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-11-03T16:15:22.0716233Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-11-03T16:15:22.0716539Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-11-03T16:15:22.0716845Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-11-03T16:15:22.0717151Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-11-03T16:15:22.0717457Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-11-03T16:15:22.0717759Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-11-03T16:15:22.0718063Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-11-03T16:15:22.0718371Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-11-03T16:15:22.0718776Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-11-03T16:15:22.0719081Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-11-03T16:15:22.0719382Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-11-03T16:15:22.0719699Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-11-03T16:15:22.0720007Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-11-03T16:15:22.0720320Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-11-03T16:15:22.0720631Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-11-03T16:15:22.0720928Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-11-03T16:15:22.0721242Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-11-03T16:15:22.0721546Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-11-03T16:15:22.0721845Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-11-03T16:15:22.0722141Z * [new branch] gh/janeyx99/310/base -> origin/gh/janeyx99/310/base 2025-11-03T16:15:22.0722429Z * [new branch] gh/janeyx99/310/head -> origin/gh/janeyx99/310/head 2025-11-03T16:15:22.0722724Z * [new branch] gh/janeyx99/310/orig -> origin/gh/janeyx99/310/orig 2025-11-03T16:15:22.0723024Z * [new branch] gh/janeyx99/311/base -> origin/gh/janeyx99/311/base 2025-11-03T16:15:22.0723363Z * [new branch] gh/janeyx99/311/head -> origin/gh/janeyx99/311/head 2025-11-03T16:15:22.0723665Z * [new branch] gh/janeyx99/311/orig -> origin/gh/janeyx99/311/orig 2025-11-03T16:15:22.0723956Z * [new branch] gh/janeyx99/312/base -> origin/gh/janeyx99/312/base 2025-11-03T16:15:22.0724253Z * [new branch] gh/janeyx99/312/head -> origin/gh/janeyx99/312/head 2025-11-03T16:15:22.0724554Z * [new branch] gh/janeyx99/312/orig -> origin/gh/janeyx99/312/orig 2025-11-03T16:15:22.0724860Z * [new branch] gh/janeyx99/313/base -> origin/gh/janeyx99/313/base 2025-11-03T16:15:22.0725158Z * [new branch] gh/janeyx99/313/head -> origin/gh/janeyx99/313/head 2025-11-03T16:15:22.0725446Z * [new branch] gh/janeyx99/313/orig -> origin/gh/janeyx99/313/orig 2025-11-03T16:15:22.0725739Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-11-03T16:15:22.0726038Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-11-03T16:15:22.0726333Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-11-03T16:15:22.0726627Z * [new branch] gh/janeyx99/315/base -> origin/gh/janeyx99/315/base 2025-11-03T16:15:22.0726916Z * [new branch] gh/janeyx99/315/head -> origin/gh/janeyx99/315/head 2025-11-03T16:15:22.0727210Z * [new branch] gh/janeyx99/315/orig -> origin/gh/janeyx99/315/orig 2025-11-03T16:15:22.0727507Z * [new branch] gh/janeyx99/316/base -> origin/gh/janeyx99/316/base 2025-11-03T16:15:22.0727803Z * [new branch] gh/janeyx99/316/head -> origin/gh/janeyx99/316/head 2025-11-03T16:15:22.0728215Z * [new branch] gh/janeyx99/316/orig -> origin/gh/janeyx99/316/orig 2025-11-03T16:15:22.0732664Z * [new branch] gh/janeyx99/317/base -> origin/gh/janeyx99/317/base 2025-11-03T16:15:22.0736921Z * [new branch] gh/janeyx99/317/head -> origin/gh/janeyx99/317/head 2025-11-03T16:15:22.0738921Z * [new branch] gh/janeyx99/317/orig -> origin/gh/janeyx99/317/orig 2025-11-03T16:15:22.0739562Z * [new branch] gh/janeyx99/318/base -> origin/gh/janeyx99/318/base 2025-11-03T16:15:22.0743849Z * [new branch] gh/janeyx99/318/head -> origin/gh/janeyx99/318/head 2025-11-03T16:15:22.0744382Z * [new branch] gh/janeyx99/318/orig -> origin/gh/janeyx99/318/orig 2025-11-03T16:15:22.0744837Z * [new branch] gh/janeyx99/319/base -> origin/gh/janeyx99/319/base 2025-11-03T16:15:22.0745671Z * [new branch] gh/janeyx99/319/head -> origin/gh/janeyx99/319/head 2025-11-03T16:15:22.0746067Z * [new branch] gh/janeyx99/319/orig -> origin/gh/janeyx99/319/orig 2025-11-03T16:15:22.0746394Z * [new branch] gh/janeyx99/320/base -> origin/gh/janeyx99/320/base 2025-11-03T16:15:22.0746726Z * [new branch] gh/janeyx99/320/head -> origin/gh/janeyx99/320/head 2025-11-03T16:15:22.0747039Z * [new branch] gh/janeyx99/320/orig -> origin/gh/janeyx99/320/orig 2025-11-03T16:15:22.0747360Z * [new branch] gh/janeyx99/321/base -> origin/gh/janeyx99/321/base 2025-11-03T16:15:22.0747675Z * [new branch] gh/janeyx99/321/head -> origin/gh/janeyx99/321/head 2025-11-03T16:15:22.0747983Z * [new branch] gh/janeyx99/321/orig -> origin/gh/janeyx99/321/orig 2025-11-03T16:15:22.0748291Z * [new branch] gh/janeyx99/322/base -> origin/gh/janeyx99/322/base 2025-11-03T16:15:22.0748596Z * [new branch] gh/janeyx99/322/head -> origin/gh/janeyx99/322/head 2025-11-03T16:15:22.0748894Z * [new branch] gh/janeyx99/322/orig -> origin/gh/janeyx99/322/orig 2025-11-03T16:15:22.0749338Z * [new branch] gh/janeyx99/323/base -> origin/gh/janeyx99/323/base 2025-11-03T16:15:22.0749660Z * [new branch] gh/janeyx99/323/head -> origin/gh/janeyx99/323/head 2025-11-03T16:15:22.0749974Z * [new branch] gh/janeyx99/323/orig -> origin/gh/janeyx99/323/orig 2025-11-03T16:15:22.0750284Z * [new branch] gh/janeyx99/324/base -> origin/gh/janeyx99/324/base 2025-11-03T16:15:22.0750587Z * [new branch] gh/janeyx99/324/head -> origin/gh/janeyx99/324/head 2025-11-03T16:15:22.0750894Z * [new branch] gh/janeyx99/324/orig -> origin/gh/janeyx99/324/orig 2025-11-03T16:15:22.0751215Z * [new branch] gh/janeyx99/325/base -> origin/gh/janeyx99/325/base 2025-11-03T16:15:22.0751531Z * [new branch] gh/janeyx99/325/head -> origin/gh/janeyx99/325/head 2025-11-03T16:15:22.0751849Z * [new branch] gh/janeyx99/325/orig -> origin/gh/janeyx99/325/orig 2025-11-03T16:15:22.0752163Z * [new branch] gh/janeyx99/326/base -> origin/gh/janeyx99/326/base 2025-11-03T16:15:22.0752480Z * [new branch] gh/janeyx99/326/head -> origin/gh/janeyx99/326/head 2025-11-03T16:15:22.0752802Z * [new branch] gh/janeyx99/326/orig -> origin/gh/janeyx99/326/orig 2025-11-03T16:15:22.0753131Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-11-03T16:15:22.0753459Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-11-03T16:15:22.0753793Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-11-03T16:15:22.0754246Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-11-03T16:15:22.0754564Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-11-03T16:15:22.0754876Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-11-03T16:15:22.0755183Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-11-03T16:15:22.0755494Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-11-03T16:15:22.0755856Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-11-03T16:15:22.0756152Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-11-03T16:15:22.0756452Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-11-03T16:15:22.0756743Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-11-03T16:15:22.0757046Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-11-03T16:15:22.0757346Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-11-03T16:15:22.0757704Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-11-03T16:15:22.0758102Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-11-03T16:15:22.0758525Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-11-03T16:15:22.0758953Z * [new branch] gh/jansel/549/base -> origin/gh/jansel/549/base 2025-11-03T16:15:22.0759377Z * [new branch] gh/jansel/549/head -> origin/gh/jansel/549/head 2025-11-03T16:15:22.0759686Z * [new branch] gh/jansel/549/orig -> origin/gh/jansel/549/orig 2025-11-03T16:15:22.0759995Z * [new branch] gh/jansel/550/base -> origin/gh/jansel/550/base 2025-11-03T16:15:22.0760292Z * [new branch] gh/jansel/550/head -> origin/gh/jansel/550/head 2025-11-03T16:15:22.0760583Z * [new branch] gh/jansel/550/orig -> origin/gh/jansel/550/orig 2025-11-03T16:15:22.0760963Z * [new branch] gh/jansel/551/base -> origin/gh/jansel/551/base 2025-11-03T16:15:22.0761504Z * [new branch] gh/jansel/551/head -> origin/gh/jansel/551/head 2025-11-03T16:15:22.0763241Z * [new branch] gh/jansel/551/orig -> origin/gh/jansel/551/orig 2025-11-03T16:15:22.0763709Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-11-03T16:15:22.0764290Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-11-03T16:15:22.0764922Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-11-03T16:15:22.0769013Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-11-03T16:15:22.0769560Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-11-03T16:15:22.0769912Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-11-03T16:15:22.0770255Z * [new branch] gh/jgong5/1/base -> origin/gh/jgong5/1/base 2025-11-03T16:15:22.0770558Z * [new branch] gh/jgong5/1/head -> origin/gh/jgong5/1/head 2025-11-03T16:15:22.0770848Z * [new branch] gh/jgong5/1/orig -> origin/gh/jgong5/1/orig 2025-11-03T16:15:22.0771310Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-11-03T16:15:22.0772134Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-11-03T16:15:22.0772523Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-11-03T16:15:22.0773074Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-11-03T16:15:22.0773480Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-11-03T16:15:22.0774123Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-11-03T16:15:22.0775648Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-11-03T16:15:22.0776260Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-11-03T16:15:22.0776956Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-11-03T16:15:22.0777384Z * [new branch] gh/jiayisunx/71/base -> origin/gh/jiayisunx/71/base 2025-11-03T16:15:22.0778010Z * [new branch] gh/jiayisunx/71/head -> origin/gh/jiayisunx/71/head 2025-11-03T16:15:22.0778680Z * [new branch] gh/jiayisunx/71/orig -> origin/gh/jiayisunx/71/orig 2025-11-03T16:15:22.0780286Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-11-03T16:15:22.0780798Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-11-03T16:15:22.0781292Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-11-03T16:15:22.0782063Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-11-03T16:15:22.0782644Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-11-03T16:15:22.0784959Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-11-03T16:15:22.0785509Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-11-03T16:15:22.0785969Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-11-03T16:15:22.0786432Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-11-03T16:15:22.0787180Z * [new branch] gh/jiayisunx/80/base -> origin/gh/jiayisunx/80/base 2025-11-03T16:15:22.0787607Z * [new branch] gh/jiayisunx/80/head -> origin/gh/jiayisunx/80/head 2025-11-03T16:15:22.0788691Z * [new branch] gh/jiayisunx/80/orig -> origin/gh/jiayisunx/80/orig 2025-11-03T16:15:22.0789270Z * [new branch] gh/jiayisunx/81/base -> origin/gh/jiayisunx/81/base 2025-11-03T16:15:22.0789796Z * [new branch] gh/jiayisunx/81/head -> origin/gh/jiayisunx/81/head 2025-11-03T16:15:22.0790480Z * [new branch] gh/jiayisunx/81/orig -> origin/gh/jiayisunx/81/orig 2025-11-03T16:15:22.0791455Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-11-03T16:15:22.0791993Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-11-03T16:15:22.0792778Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-11-03T16:15:22.0793689Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-11-03T16:15:22.0794153Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-11-03T16:15:22.0795147Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-11-03T16:15:22.0795875Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-11-03T16:15:22.0796620Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-11-03T16:15:22.0797208Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-11-03T16:15:22.0798746Z * [new branch] gh/jiayisunx/85/base -> origin/gh/jiayisunx/85/base 2025-11-03T16:15:22.0799069Z * [new branch] gh/jiayisunx/85/head -> origin/gh/jiayisunx/85/head 2025-11-03T16:15:22.0799478Z * [new branch] gh/jiayisunx/85/orig -> origin/gh/jiayisunx/85/orig 2025-11-03T16:15:22.0800138Z * [new branch] gh/jiayisunx/86/base -> origin/gh/jiayisunx/86/base 2025-11-03T16:15:22.0800941Z * [new branch] gh/jiayisunx/86/head -> origin/gh/jiayisunx/86/head 2025-11-03T16:15:22.0801425Z * [new branch] gh/jiayisunx/86/orig -> origin/gh/jiayisunx/86/orig 2025-11-03T16:15:22.0802863Z * [new branch] gh/jiayisunx/87/base -> origin/gh/jiayisunx/87/base 2025-11-03T16:15:22.0803388Z * [new branch] gh/jiayisunx/87/head -> origin/gh/jiayisunx/87/head 2025-11-03T16:15:22.0803817Z * [new branch] gh/jiayisunx/87/orig -> origin/gh/jiayisunx/87/orig 2025-11-03T16:15:22.0805267Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-11-03T16:15:22.0805832Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-11-03T16:15:22.0807719Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-11-03T16:15:22.0808120Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-11-03T16:15:22.0808698Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-11-03T16:15:22.0811167Z * [new branch] gh/karthickai/7/base -> origin/gh/karthickai/7/base 2025-11-03T16:15:22.0811703Z * [new branch] gh/karthickai/7/head -> origin/gh/karthickai/7/head 2025-11-03T16:15:22.0812163Z * [new branch] gh/karthickai/7/orig -> origin/gh/karthickai/7/orig 2025-11-03T16:15:22.0812976Z * [new branch] gh/karthickai/8/base -> origin/gh/karthickai/8/base 2025-11-03T16:15:22.0813501Z * [new branch] gh/karthickai/8/head -> origin/gh/karthickai/8/head 2025-11-03T16:15:22.0813842Z * [new branch] gh/karthickai/8/orig -> origin/gh/karthickai/8/orig 2025-11-03T16:15:22.0814500Z * [new branch] gh/karthickai/9/base -> origin/gh/karthickai/9/base 2025-11-03T16:15:22.0815036Z * [new branch] gh/karthickai/9/head -> origin/gh/karthickai/9/head 2025-11-03T16:15:22.0815826Z * [new branch] gh/karthickai/9/orig -> origin/gh/karthickai/9/orig 2025-11-03T16:15:22.0821202Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-11-03T16:15:22.0821741Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-11-03T16:15:22.0822218Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-11-03T16:15:22.0823056Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-11-03T16:15:22.0823450Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-11-03T16:15:22.0823788Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-11-03T16:15:22.0824119Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-11-03T16:15:22.0824449Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-11-03T16:15:22.0824789Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-11-03T16:15:22.0825282Z * [new branch] gh/kurtamohler/55/base -> origin/gh/kurtamohler/55/base 2025-11-03T16:15:22.0825740Z * [new branch] gh/kurtamohler/55/head -> origin/gh/kurtamohler/55/head 2025-11-03T16:15:22.0826208Z * [new branch] gh/kurtamohler/55/orig -> origin/gh/kurtamohler/55/orig 2025-11-03T16:15:22.0827032Z * [new branch] gh/kurtamohler/56/base -> origin/gh/kurtamohler/56/base 2025-11-03T16:15:22.0827416Z * [new branch] gh/kurtamohler/56/head -> origin/gh/kurtamohler/56/head 2025-11-03T16:15:22.0827746Z * [new branch] gh/kurtamohler/56/orig -> origin/gh/kurtamohler/56/orig 2025-11-03T16:15:22.0828102Z * [new branch] gh/kurtamohler/57/base -> origin/gh/kurtamohler/57/base 2025-11-03T16:15:22.0828613Z * [new branch] gh/kurtamohler/57/head -> origin/gh/kurtamohler/57/head 2025-11-03T16:15:22.0829289Z * [new branch] gh/kurtamohler/57/orig -> origin/gh/kurtamohler/57/orig 2025-11-03T16:15:22.0830442Z * [new branch] gh/kurtamohler/58/base -> origin/gh/kurtamohler/58/base 2025-11-03T16:15:22.0831096Z * [new branch] gh/kurtamohler/58/head -> origin/gh/kurtamohler/58/head 2025-11-03T16:15:22.0831535Z * [new branch] gh/kurtamohler/58/orig -> origin/gh/kurtamohler/58/orig 2025-11-03T16:15:22.0833993Z * [new branch] gh/kurtamohler/59/base -> origin/gh/kurtamohler/59/base 2025-11-03T16:15:22.0834408Z * [new branch] gh/kurtamohler/59/head -> origin/gh/kurtamohler/59/head 2025-11-03T16:15:22.0834762Z * [new branch] gh/kurtamohler/59/orig -> origin/gh/kurtamohler/59/orig 2025-11-03T16:15:22.0835268Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-11-03T16:15:22.0835752Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-11-03T16:15:22.0836390Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-11-03T16:15:22.0838687Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-11-03T16:15:22.0839231Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-11-03T16:15:22.0839688Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-11-03T16:15:22.0840017Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-11-03T16:15:22.0840560Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-11-03T16:15:22.0844944Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-11-03T16:15:22.0845472Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-11-03T16:15:22.0846061Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-11-03T16:15:22.0846978Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-11-03T16:15:22.0847416Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-11-03T16:15:22.0847784Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-11-03T16:15:22.0848112Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-11-03T16:15:22.0848463Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-11-03T16:15:22.0848824Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-11-03T16:15:22.0849172Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-11-03T16:15:22.0849526Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-11-03T16:15:22.0850015Z * [new branch] gh/kwen2501/231/base -> origin/gh/kwen2501/231/base 2025-11-03T16:15:22.0850345Z * [new branch] gh/kwen2501/231/head -> origin/gh/kwen2501/231/head 2025-11-03T16:15:22.0850905Z * [new branch] gh/kwen2501/231/orig -> origin/gh/kwen2501/231/orig 2025-11-03T16:15:22.0852560Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-11-03T16:15:22.0852890Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-11-03T16:15:22.0853220Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-11-03T16:15:22.0855374Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-11-03T16:15:22.0855905Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-11-03T16:15:22.0856359Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-11-03T16:15:22.0856673Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-11-03T16:15:22.0857188Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-11-03T16:15:22.0857913Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-11-03T16:15:22.0859012Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-11-03T16:15:22.0859394Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-11-03T16:15:22.0861682Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-11-03T16:15:22.0862200Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-11-03T16:15:22.0862637Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-11-03T16:15:22.0863417Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-11-03T16:15:22.0863945Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-11-03T16:15:22.0864387Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-11-03T16:15:22.0864720Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-11-03T16:15:22.0865840Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-11-03T16:15:22.0866249Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-11-03T16:15:22.0866887Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-11-03T16:15:22.0868437Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-11-03T16:15:22.0868979Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-11-03T16:15:22.0869592Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-11-03T16:15:22.0870325Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-11-03T16:15:22.0870873Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-11-03T16:15:22.0871506Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-11-03T16:15:22.0872635Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-11-03T16:15:22.0872949Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-11-03T16:15:22.0873533Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-11-03T16:15:22.0875223Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-11-03T16:15:22.0875765Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-11-03T16:15:22.0876240Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-11-03T16:15:22.0876736Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-11-03T16:15:22.0877462Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-11-03T16:15:22.0878332Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-11-03T16:15:22.0879267Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-11-03T16:15:22.0879709Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-11-03T16:15:22.0880350Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-11-03T16:15:22.0884104Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-11-03T16:15:22.0884675Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-11-03T16:15:22.0885142Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-11-03T16:15:22.0885997Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-11-03T16:15:22.0886521Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-11-03T16:15:22.0886844Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-11-03T16:15:22.0887155Z * [new branch] gh/kwen2501/272/base -> origin/gh/kwen2501/272/base 2025-11-03T16:15:22.0887462Z * [new branch] gh/kwen2501/272/head -> origin/gh/kwen2501/272/head 2025-11-03T16:15:22.0887773Z * [new branch] gh/kwen2501/272/orig -> origin/gh/kwen2501/272/orig 2025-11-03T16:15:22.0888203Z * [new branch] gh/kwen2501/273/base -> origin/gh/kwen2501/273/base 2025-11-03T16:15:22.0888987Z * [new branch] gh/kwen2501/273/head -> origin/gh/kwen2501/273/head 2025-11-03T16:15:22.0889764Z * [new branch] gh/kwen2501/273/orig -> origin/gh/kwen2501/273/orig 2025-11-03T16:15:22.0890273Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-11-03T16:15:22.0890941Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-11-03T16:15:22.0891547Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-11-03T16:15:22.0894038Z * [new branch] gh/kwen2501/275/base -> origin/gh/kwen2501/275/base 2025-11-03T16:15:22.0894570Z * [new branch] gh/kwen2501/275/head -> origin/gh/kwen2501/275/head 2025-11-03T16:15:22.0895010Z * [new branch] gh/kwen2501/275/orig -> origin/gh/kwen2501/275/orig 2025-11-03T16:15:22.0895772Z * [new branch] gh/kwen2501/276/base -> origin/gh/kwen2501/276/base 2025-11-03T16:15:22.0896295Z * [new branch] gh/kwen2501/276/head -> origin/gh/kwen2501/276/head 2025-11-03T16:15:22.0896615Z * [new branch] gh/kwen2501/276/orig -> origin/gh/kwen2501/276/orig 2025-11-03T16:15:22.0897112Z * [new branch] gh/kwen2501/277/base -> origin/gh/kwen2501/277/base 2025-11-03T16:15:22.0897427Z * [new branch] gh/kwen2501/277/head -> origin/gh/kwen2501/277/head 2025-11-03T16:15:22.0897949Z * [new branch] gh/kwen2501/277/orig -> origin/gh/kwen2501/277/orig 2025-11-03T16:15:22.0899778Z * [new branch] gh/kwen2501/278/base -> origin/gh/kwen2501/278/base 2025-11-03T16:15:22.0900304Z * [new branch] gh/kwen2501/278/head -> origin/gh/kwen2501/278/head 2025-11-03T16:15:22.0900742Z * [new branch] gh/kwen2501/278/orig -> origin/gh/kwen2501/278/orig 2025-11-03T16:15:22.0901267Z * [new branch] gh/kwen2501/279/base -> origin/gh/kwen2501/279/base 2025-11-03T16:15:22.0901866Z * [new branch] gh/kwen2501/279/head -> origin/gh/kwen2501/279/head 2025-11-03T16:15:22.0902849Z * [new branch] gh/kwen2501/279/orig -> origin/gh/kwen2501/279/orig 2025-11-03T16:15:22.0906099Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-11-03T16:15:22.0906640Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-11-03T16:15:22.0907100Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-11-03T16:15:22.0907920Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-11-03T16:15:22.0908315Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-11-03T16:15:22.0908653Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-11-03T16:15:22.0909169Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-11-03T16:15:22.0909520Z * [new branch] gh/laithsakka/285/base -> origin/gh/laithsakka/285/base 2025-11-03T16:15:22.0909870Z * [new branch] gh/laithsakka/285/head -> origin/gh/laithsakka/285/head 2025-11-03T16:15:22.0910504Z * [new branch] gh/laithsakka/285/orig -> origin/gh/laithsakka/285/orig 2025-11-03T16:15:22.0911834Z * [new branch] gh/laithsakka/287/base -> origin/gh/laithsakka/287/base 2025-11-03T16:15:22.0912166Z * [new branch] gh/laithsakka/287/head -> origin/gh/laithsakka/287/head 2025-11-03T16:15:22.0912681Z * [new branch] gh/laithsakka/287/orig -> origin/gh/laithsakka/287/orig 2025-11-03T16:15:22.0914636Z * [new branch] gh/laithsakka/289/base -> origin/gh/laithsakka/289/base 2025-11-03T16:15:22.0915185Z * [new branch] gh/laithsakka/289/head -> origin/gh/laithsakka/289/head 2025-11-03T16:15:22.0915736Z * [new branch] gh/laithsakka/289/orig -> origin/gh/laithsakka/289/orig 2025-11-03T16:15:22.0918792Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-11-03T16:15:22.0924218Z * [new branch] gh/laithsakka/292/base -> origin/gh/laithsakka/292/base 2025-11-03T16:15:22.0924765Z * [new branch] gh/laithsakka/292/head -> origin/gh/laithsakka/292/head 2025-11-03T16:15:22.0925237Z * [new branch] gh/laithsakka/292/orig -> origin/gh/laithsakka/292/orig 2025-11-03T16:15:22.0925581Z * [new branch] gh/laithsakka/293/base -> origin/gh/laithsakka/293/base 2025-11-03T16:15:22.0928238Z * [new branch] gh/laithsakka/293/head -> origin/gh/laithsakka/293/head 2025-11-03T16:15:22.0928802Z * [new branch] gh/laithsakka/293/orig -> origin/gh/laithsakka/293/orig 2025-11-03T16:15:22.0929281Z * [new branch] gh/laithsakka/294/base -> origin/gh/laithsakka/294/base 2025-11-03T16:15:22.0930063Z * [new branch] gh/laithsakka/294/head -> origin/gh/laithsakka/294/head 2025-11-03T16:15:22.0930247Z * [new branch] gh/laithsakka/294/orig -> origin/gh/laithsakka/294/orig 2025-11-03T16:15:22.0930387Z * [new branch] gh/laithsakka/295/base -> origin/gh/laithsakka/295/base 2025-11-03T16:15:22.0930524Z * [new branch] gh/laithsakka/295/head -> origin/gh/laithsakka/295/head 2025-11-03T16:15:22.0930650Z * [new branch] gh/laithsakka/295/orig -> origin/gh/laithsakka/295/orig 2025-11-03T16:15:22.0931200Z * [new branch] gh/laithsakka/296/base -> origin/gh/laithsakka/296/base 2025-11-03T16:15:22.0931330Z * [new branch] gh/laithsakka/296/head -> origin/gh/laithsakka/296/head 2025-11-03T16:15:22.0931458Z * [new branch] gh/laithsakka/296/orig -> origin/gh/laithsakka/296/orig 2025-11-03T16:15:22.0931596Z * [new branch] gh/laithsakka/297/base -> origin/gh/laithsakka/297/base 2025-11-03T16:15:22.0931721Z * [new branch] gh/laithsakka/297/head -> origin/gh/laithsakka/297/head 2025-11-03T16:15:22.0931993Z * [new branch] gh/laithsakka/297/orig -> origin/gh/laithsakka/297/orig 2025-11-03T16:15:22.0938494Z * [new branch] gh/laithsakka/298/base -> origin/gh/laithsakka/298/base 2025-11-03T16:15:22.0938807Z * [new branch] gh/laithsakka/298/head -> origin/gh/laithsakka/298/head 2025-11-03T16:15:22.0939036Z * [new branch] gh/laithsakka/298/orig -> origin/gh/laithsakka/298/orig 2025-11-03T16:15:22.0939182Z * [new branch] gh/laithsakka/299/base -> origin/gh/laithsakka/299/base 2025-11-03T16:15:22.0939412Z * [new branch] gh/laithsakka/299/head -> origin/gh/laithsakka/299/head 2025-11-03T16:15:22.0939565Z * [new branch] gh/laithsakka/299/orig -> origin/gh/laithsakka/299/orig 2025-11-03T16:15:22.0939728Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-11-03T16:15:22.0939863Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-11-03T16:15:22.0942436Z * [new branch] gh/laithsakka/300/base -> origin/gh/laithsakka/300/base 2025-11-03T16:15:22.0942762Z * [new branch] gh/laithsakka/300/head -> origin/gh/laithsakka/300/head 2025-11-03T16:15:22.0942984Z * [new branch] gh/laithsakka/300/orig -> origin/gh/laithsakka/300/orig 2025-11-03T16:15:22.0943147Z * [new branch] gh/laithsakka/301/base -> origin/gh/laithsakka/301/base 2025-11-03T16:15:22.0943276Z * [new branch] gh/laithsakka/301/head -> origin/gh/laithsakka/301/head 2025-11-03T16:15:22.0943531Z * [new branch] gh/laithsakka/301/orig -> origin/gh/laithsakka/301/orig 2025-11-03T16:15:22.0947692Z * [new branch] gh/laithsakka/302/base -> origin/gh/laithsakka/302/base 2025-11-03T16:15:22.0948030Z * [new branch] gh/laithsakka/302/head -> origin/gh/laithsakka/302/head 2025-11-03T16:15:22.0948275Z * [new branch] gh/laithsakka/302/orig -> origin/gh/laithsakka/302/orig 2025-11-03T16:15:22.0948442Z * [new branch] gh/laithsakka/303/base -> origin/gh/laithsakka/303/base 2025-11-03T16:15:22.0949067Z * [new branch] gh/laithsakka/303/head -> origin/gh/laithsakka/303/head 2025-11-03T16:15:22.0949242Z * [new branch] gh/laithsakka/303/orig -> origin/gh/laithsakka/303/orig 2025-11-03T16:15:22.0949393Z * [new branch] gh/laithsakka/304/base -> origin/gh/laithsakka/304/base 2025-11-03T16:15:22.0949529Z * [new branch] gh/laithsakka/304/head -> origin/gh/laithsakka/304/head 2025-11-03T16:15:22.0949662Z * [new branch] gh/laithsakka/304/orig -> origin/gh/laithsakka/304/orig 2025-11-03T16:15:22.0949952Z * [new branch] gh/laithsakka/305/base -> origin/gh/laithsakka/305/base 2025-11-03T16:15:22.0950091Z * [new branch] gh/laithsakka/305/head -> origin/gh/laithsakka/305/head 2025-11-03T16:15:22.0950242Z * [new branch] gh/laithsakka/305/orig -> origin/gh/laithsakka/305/orig 2025-11-03T16:15:22.0950385Z * [new branch] gh/laithsakka/306/base -> origin/gh/laithsakka/306/base 2025-11-03T16:15:22.0950530Z * [new branch] gh/laithsakka/306/head -> origin/gh/laithsakka/306/head 2025-11-03T16:15:22.0950669Z * [new branch] gh/laithsakka/306/orig -> origin/gh/laithsakka/306/orig 2025-11-03T16:15:22.0953165Z * [new branch] gh/laithsakka/307/base -> origin/gh/laithsakka/307/base 2025-11-03T16:15:22.0953336Z * [new branch] gh/laithsakka/307/head -> origin/gh/laithsakka/307/head 2025-11-03T16:15:22.0953485Z * [new branch] gh/laithsakka/307/orig -> origin/gh/laithsakka/307/orig 2025-11-03T16:15:22.0953644Z * [new branch] gh/laithsakka/308/base -> origin/gh/laithsakka/308/base 2025-11-03T16:15:22.0954220Z * [new branch] gh/laithsakka/308/head -> origin/gh/laithsakka/308/head 2025-11-03T16:15:22.0955232Z * [new branch] gh/laithsakka/308/orig -> origin/gh/laithsakka/308/orig 2025-11-03T16:15:22.0957849Z * [new branch] gh/laithsakka/309/base -> origin/gh/laithsakka/309/base 2025-11-03T16:15:22.0958165Z * [new branch] gh/laithsakka/309/head -> origin/gh/laithsakka/309/head 2025-11-03T16:15:22.0958390Z * [new branch] gh/laithsakka/309/orig -> origin/gh/laithsakka/309/orig 2025-11-03T16:15:22.0958554Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-11-03T16:15:22.0958791Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-11-03T16:15:22.0960180Z * [new branch] gh/laithsakka/310/base -> origin/gh/laithsakka/310/base 2025-11-03T16:15:22.0960451Z * [new branch] gh/laithsakka/310/head -> origin/gh/laithsakka/310/head 2025-11-03T16:15:22.0960771Z * [new branch] gh/laithsakka/310/orig -> origin/gh/laithsakka/310/orig 2025-11-03T16:15:22.0963251Z * [new branch] gh/laithsakka/311/base -> origin/gh/laithsakka/311/base 2025-11-03T16:15:22.0963569Z * [new branch] gh/laithsakka/311/head -> origin/gh/laithsakka/311/head 2025-11-03T16:15:22.0963797Z * [new branch] gh/laithsakka/311/orig -> origin/gh/laithsakka/311/orig 2025-11-03T16:15:22.0963948Z * [new branch] gh/laithsakka/312/base -> origin/gh/laithsakka/312/base 2025-11-03T16:15:22.0965265Z * [new branch] gh/laithsakka/312/head -> origin/gh/laithsakka/312/head 2025-11-03T16:15:22.0965578Z * [new branch] gh/laithsakka/312/orig -> origin/gh/laithsakka/312/orig 2025-11-03T16:15:22.0967645Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-11-03T16:15:22.0967978Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-11-03T16:15:22.0968218Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-11-03T16:15:22.0968377Z * [new branch] gh/laithsakka/314/base -> origin/gh/laithsakka/314/base 2025-11-03T16:15:22.0972149Z * [new branch] gh/laithsakka/314/head -> origin/gh/laithsakka/314/head 2025-11-03T16:15:22.0972485Z * [new branch] gh/laithsakka/314/orig -> origin/gh/laithsakka/314/orig 2025-11-03T16:15:22.0972717Z * [new branch] gh/laithsakka/315/base -> origin/gh/laithsakka/315/base 2025-11-03T16:15:22.0972881Z * [new branch] gh/laithsakka/315/head -> origin/gh/laithsakka/315/head 2025-11-03T16:15:22.0973166Z * [new branch] gh/laithsakka/315/orig -> origin/gh/laithsakka/315/orig 2025-11-03T16:15:22.0973463Z * [new branch] gh/laithsakka/316/base -> origin/gh/laithsakka/316/base 2025-11-03T16:15:22.0973992Z * [new branch] gh/laithsakka/316/head -> origin/gh/laithsakka/316/head 2025-11-03T16:15:22.0974377Z * [new branch] gh/laithsakka/316/orig -> origin/gh/laithsakka/316/orig 2025-11-03T16:15:22.0978432Z * [new branch] gh/laithsakka/317/base -> origin/gh/laithsakka/317/base 2025-11-03T16:15:22.0978758Z * [new branch] gh/laithsakka/317/head -> origin/gh/laithsakka/317/head 2025-11-03T16:15:22.0978989Z * [new branch] gh/laithsakka/317/orig -> origin/gh/laithsakka/317/orig 2025-11-03T16:15:22.0979160Z * [new branch] gh/laithsakka/318/base -> origin/gh/laithsakka/318/base 2025-11-03T16:15:22.0979413Z * [new branch] gh/laithsakka/318/head -> origin/gh/laithsakka/318/head 2025-11-03T16:15:22.0980059Z * [new branch] gh/laithsakka/318/orig -> origin/gh/laithsakka/318/orig 2025-11-03T16:15:22.0980240Z * [new branch] gh/laithsakka/319/base -> origin/gh/laithsakka/319/base 2025-11-03T16:15:22.0980387Z * [new branch] gh/laithsakka/319/head -> origin/gh/laithsakka/319/head 2025-11-03T16:15:22.0980938Z * [new branch] gh/laithsakka/319/orig -> origin/gh/laithsakka/319/orig 2025-11-03T16:15:22.0984973Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-11-03T16:15:22.0985288Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-11-03T16:15:22.0985514Z * [new branch] gh/laithsakka/320/base -> origin/gh/laithsakka/320/base 2025-11-03T16:15:22.0985679Z * [new branch] gh/laithsakka/320/head -> origin/gh/laithsakka/320/head 2025-11-03T16:15:22.0985813Z * [new branch] gh/laithsakka/320/orig -> origin/gh/laithsakka/320/orig 2025-11-03T16:15:22.0986095Z * [new branch] gh/laithsakka/321/base -> origin/gh/laithsakka/321/base 2025-11-03T16:15:22.0986694Z * [new branch] gh/laithsakka/321/head -> origin/gh/laithsakka/321/head 2025-11-03T16:15:22.0987215Z * [new branch] gh/laithsakka/321/orig -> origin/gh/laithsakka/321/orig 2025-11-03T16:15:22.0988936Z * [new branch] gh/liangel-02/1/base -> origin/gh/liangel-02/1/base 2025-11-03T16:15:22.0989249Z * [new branch] gh/liangel-02/1/head -> origin/gh/liangel-02/1/head 2025-11-03T16:15:22.0989499Z * [new branch] gh/liangel-02/1/orig -> origin/gh/liangel-02/1/orig 2025-11-03T16:15:22.0991751Z * [new branch] gh/liangel-02/2/base -> origin/gh/liangel-02/2/base 2025-11-03T16:15:22.0991902Z * [new branch] gh/liangel-02/2/head -> origin/gh/liangel-02/2/head 2025-11-03T16:15:22.0992061Z * [new branch] gh/liangel-02/2/orig -> origin/gh/liangel-02/2/orig 2025-11-03T16:15:22.0996416Z * [new branch] gh/liangel/1/base -> origin/gh/liangel/1/base 2025-11-03T16:15:22.0996732Z * [new branch] gh/liangel/1/head -> origin/gh/liangel/1/head 2025-11-03T16:15:22.0996959Z * [new branch] gh/liangel/1/orig -> origin/gh/liangel/1/orig 2025-11-03T16:15:22.0997098Z * [new branch] gh/liangel/2/base -> origin/gh/liangel/2/base 2025-11-03T16:15:22.0997296Z * [new branch] gh/liangel/2/head -> origin/gh/liangel/2/head 2025-11-03T16:15:22.0997434Z * [new branch] gh/liangel/2/orig -> origin/gh/liangel/2/orig 2025-11-03T16:15:22.0998386Z * [new branch] gh/liangel/3/base -> origin/gh/liangel/3/base 2025-11-03T16:15:22.0998746Z * [new branch] gh/liangel/3/head -> origin/gh/liangel/3/head 2025-11-03T16:15:22.1001040Z * [new branch] gh/liangel/3/orig -> origin/gh/liangel/3/orig 2025-11-03T16:15:22.1001372Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-11-03T16:15:22.1001574Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-11-03T16:15:22.1001792Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-11-03T16:15:22.1006866Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-11-03T16:15:22.1007192Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-11-03T16:15:22.1007425Z * [new branch] gh/lucaskabela/10/base -> origin/gh/lucaskabela/10/base 2025-11-03T16:15:22.1007580Z * [new branch] gh/lucaskabela/10/head -> origin/gh/lucaskabela/10/head 2025-11-03T16:15:22.1008078Z * [new branch] gh/lucaskabela/10/orig -> origin/gh/lucaskabela/10/orig 2025-11-03T16:15:22.1012304Z * [new branch] gh/lucaskabela/2/base -> origin/gh/lucaskabela/2/base 2025-11-03T16:15:22.1012631Z * [new branch] gh/lucaskabela/2/head -> origin/gh/lucaskabela/2/head 2025-11-03T16:15:22.1012858Z * [new branch] gh/lucaskabela/3/base -> origin/gh/lucaskabela/3/base 2025-11-03T16:15:22.1013006Z * [new branch] gh/lucaskabela/3/head -> origin/gh/lucaskabela/3/head 2025-11-03T16:15:22.1013367Z * [new branch] gh/lucaskabela/3/orig -> origin/gh/lucaskabela/3/orig 2025-11-03T16:15:22.1013965Z * [new branch] gh/lucaskabela/4/base -> origin/gh/lucaskabela/4/base 2025-11-03T16:15:22.1014122Z * [new branch] gh/lucaskabela/4/head -> origin/gh/lucaskabela/4/head 2025-11-03T16:15:22.1014258Z * [new branch] gh/lucaskabela/4/orig -> origin/gh/lucaskabela/4/orig 2025-11-03T16:15:22.1014699Z * [new branch] gh/lucaskabela/5/base -> origin/gh/lucaskabela/5/base 2025-11-03T16:15:22.1015621Z * [new branch] gh/lucaskabela/5/head -> origin/gh/lucaskabela/5/head 2025-11-03T16:15:22.1017833Z * [new branch] gh/lucaskabela/5/orig -> origin/gh/lucaskabela/5/orig 2025-11-03T16:15:22.1018381Z * [new branch] gh/lucaskabela/6/base -> origin/gh/lucaskabela/6/base 2025-11-03T16:15:22.1018619Z * [new branch] gh/lucaskabela/6/head -> origin/gh/lucaskabela/6/head 2025-11-03T16:15:22.1018766Z * [new branch] gh/lucaskabela/6/orig -> origin/gh/lucaskabela/6/orig 2025-11-03T16:15:22.1018904Z * [new branch] gh/lucaskabela/7/base -> origin/gh/lucaskabela/7/base 2025-11-03T16:15:22.1019761Z * [new branch] gh/lucaskabela/7/head -> origin/gh/lucaskabela/7/head 2025-11-03T16:15:22.1020178Z * [new branch] gh/lucaskabela/7/orig -> origin/gh/lucaskabela/7/orig 2025-11-03T16:15:22.1022814Z * [new branch] gh/lucaskabela/8/base -> origin/gh/lucaskabela/8/base 2025-11-03T16:15:22.1023139Z * [new branch] gh/lucaskabela/8/head -> origin/gh/lucaskabela/8/head 2025-11-03T16:15:22.1023364Z * [new branch] gh/lucaskabela/8/orig -> origin/gh/lucaskabela/8/orig 2025-11-03T16:15:22.1023529Z * [new branch] gh/lucaskabela/9/base -> origin/gh/lucaskabela/9/base 2025-11-03T16:15:22.1023763Z * [new branch] gh/lucaskabela/9/head -> origin/gh/lucaskabela/9/head 2025-11-03T16:15:22.1024120Z * [new branch] gh/lucaskabela/9/orig -> origin/gh/lucaskabela/9/orig 2025-11-03T16:15:22.1028006Z * [new branch] gh/lw/10/base -> origin/gh/lw/10/base 2025-11-03T16:15:22.1028290Z * [new branch] gh/lw/10/head -> origin/gh/lw/10/head 2025-11-03T16:15:22.1028523Z * [new branch] gh/lw/10/orig -> origin/gh/lw/10/orig 2025-11-03T16:15:22.1028958Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-11-03T16:15:22.1029097Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-11-03T16:15:22.1029338Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-11-03T16:15:22.1029810Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-11-03T16:15:22.1030281Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-11-03T16:15:22.1031355Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-11-03T16:15:22.1031814Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-11-03T16:15:22.1032702Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-11-03T16:15:22.1033163Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-11-03T16:15:22.1038420Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-11-03T16:15:22.1038732Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-11-03T16:15:22.1038932Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-11-03T16:15:22.1039064Z * [new branch] gh/lw/7/base -> origin/gh/lw/7/base 2025-11-03T16:15:22.1039275Z * [new branch] gh/lw/7/head -> origin/gh/lw/7/head 2025-11-03T16:15:22.1039396Z * [new branch] gh/lw/7/orig -> origin/gh/lw/7/orig 2025-11-03T16:15:22.1039604Z * [new branch] gh/lw/8/base -> origin/gh/lw/8/base 2025-11-03T16:15:22.1040284Z * [new branch] gh/lw/8/head -> origin/gh/lw/8/head 2025-11-03T16:15:22.1040422Z * [new branch] gh/lw/8/orig -> origin/gh/lw/8/orig 2025-11-03T16:15:22.1040824Z * [new branch] gh/lw/9/base -> origin/gh/lw/9/base 2025-11-03T16:15:22.1042578Z * [new branch] gh/lw/9/head -> origin/gh/lw/9/head 2025-11-03T16:15:22.1042873Z * [new branch] gh/lw/9/orig -> origin/gh/lw/9/orig 2025-11-03T16:15:22.1043283Z * [new branch] gh/maggiemoss/1/base -> origin/gh/maggiemoss/1/base 2025-11-03T16:15:22.1045561Z * [new branch] gh/maggiemoss/1/head -> origin/gh/maggiemoss/1/head 2025-11-03T16:15:22.1045884Z * [new branch] gh/maggiemoss/1/orig -> origin/gh/maggiemoss/1/orig 2025-11-03T16:15:22.1046109Z * [new branch] gh/maggiemoss/2/base -> origin/gh/maggiemoss/2/base 2025-11-03T16:15:22.1046270Z * [new branch] gh/maggiemoss/2/head -> origin/gh/maggiemoss/2/head 2025-11-03T16:15:22.1046764Z * [new branch] gh/maggiemoss/2/orig -> origin/gh/maggiemoss/2/orig 2025-11-03T16:15:22.1050695Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-11-03T16:15:22.1051037Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-11-03T16:15:22.1051255Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-11-03T16:15:22.1051400Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-11-03T16:15:22.1051626Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-11-03T16:15:22.1051767Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-11-03T16:15:22.1052340Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-11-03T16:15:22.1053278Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-11-03T16:15:22.1053511Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-11-03T16:15:22.1056280Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-11-03T16:15:22.1056747Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-11-03T16:15:22.1057018Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-11-03T16:15:22.1057171Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-11-03T16:15:22.1057437Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-11-03T16:15:22.1058541Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-11-03T16:15:22.1058889Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-11-03T16:15:22.1061398Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-11-03T16:15:22.1061705Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-11-03T16:15:22.1061912Z * [new branch] gh/malfet/547/base -> origin/gh/malfet/547/base 2025-11-03T16:15:22.1062075Z * [new branch] gh/malfet/547/head -> origin/gh/malfet/547/head 2025-11-03T16:15:22.1062247Z * [new branch] gh/malfet/547/orig -> origin/gh/malfet/547/orig 2025-11-03T16:15:22.1064016Z * [new branch] gh/malfet/548/base -> origin/gh/malfet/548/base 2025-11-03T16:15:22.1064337Z * [new branch] gh/malfet/548/head -> origin/gh/malfet/548/head 2025-11-03T16:15:22.1064555Z * [new branch] gh/malfet/548/orig -> origin/gh/malfet/548/orig 2025-11-03T16:15:22.1066928Z * [new branch] gh/malfet/549/base -> origin/gh/malfet/549/base 2025-11-03T16:15:22.1067231Z * [new branch] gh/malfet/549/head -> origin/gh/malfet/549/head 2025-11-03T16:15:22.1067441Z * [new branch] gh/malfet/549/orig -> origin/gh/malfet/549/orig 2025-11-03T16:15:22.1067644Z * [new branch] gh/malfet/550/base -> origin/gh/malfet/550/base 2025-11-03T16:15:22.1067799Z * [new branch] gh/malfet/550/head -> origin/gh/malfet/550/head 2025-11-03T16:15:22.1068899Z * [new branch] gh/malfet/550/orig -> origin/gh/malfet/550/orig 2025-11-03T16:15:22.1069402Z * [new branch] gh/malfet/551/base -> origin/gh/malfet/551/base 2025-11-03T16:15:22.1070319Z * [new branch] gh/malfet/551/head -> origin/gh/malfet/551/head 2025-11-03T16:15:22.1070872Z * [new branch] gh/malfet/551/orig -> origin/gh/malfet/551/orig 2025-11-03T16:15:22.1072251Z * [new branch] gh/malfet/552/base -> origin/gh/malfet/552/base 2025-11-03T16:15:22.1072558Z * [new branch] gh/malfet/552/head -> origin/gh/malfet/552/head 2025-11-03T16:15:22.1073533Z * [new branch] gh/malfet/552/orig -> origin/gh/malfet/552/orig 2025-11-03T16:15:22.1074150Z * [new branch] gh/malfet/553/base -> origin/gh/malfet/553/base 2025-11-03T16:15:22.1075387Z * [new branch] gh/malfet/553/head -> origin/gh/malfet/553/head 2025-11-03T16:15:22.1075657Z * [new branch] gh/malfet/553/orig -> origin/gh/malfet/553/orig 2025-11-03T16:15:22.1078816Z * [new branch] gh/malfet/554/base -> origin/gh/malfet/554/base 2025-11-03T16:15:22.1078987Z * [new branch] gh/malfet/554/head -> origin/gh/malfet/554/head 2025-11-03T16:15:22.1079119Z * [new branch] gh/malfet/554/orig -> origin/gh/malfet/554/orig 2025-11-03T16:15:22.1079286Z * [new branch] gh/malfet/555/base -> origin/gh/malfet/555/base 2025-11-03T16:15:22.1079584Z * [new branch] gh/malfet/555/head -> origin/gh/malfet/555/head 2025-11-03T16:15:22.1079896Z * [new branch] gh/malfet/555/orig -> origin/gh/malfet/555/orig 2025-11-03T16:15:22.1084095Z * [new branch] gh/malfet/556/base -> origin/gh/malfet/556/base 2025-11-03T16:15:22.1084575Z * [new branch] gh/malfet/556/head -> origin/gh/malfet/556/head 2025-11-03T16:15:22.1084853Z * [new branch] gh/malfet/556/orig -> origin/gh/malfet/556/orig 2025-11-03T16:15:22.1085091Z * [new branch] gh/malfet/557/base -> origin/gh/malfet/557/base 2025-11-03T16:15:22.1085243Z * [new branch] gh/malfet/557/head -> origin/gh/malfet/557/head 2025-11-03T16:15:22.1085874Z * [new branch] gh/malfet/557/orig -> origin/gh/malfet/557/orig 2025-11-03T16:15:22.1086030Z * [new branch] gh/malfet/558/base -> origin/gh/malfet/558/base 2025-11-03T16:15:22.1086295Z * [new branch] gh/malfet/558/head -> origin/gh/malfet/558/head 2025-11-03T16:15:22.1087272Z * [new branch] gh/malfet/558/orig -> origin/gh/malfet/558/orig 2025-11-03T16:15:22.1090516Z * [new branch] gh/malfet/559/base -> origin/gh/malfet/559/base 2025-11-03T16:15:22.1090858Z * [new branch] gh/malfet/559/head -> origin/gh/malfet/559/head 2025-11-03T16:15:22.1091100Z * [new branch] gh/malfet/559/orig -> origin/gh/malfet/559/orig 2025-11-03T16:15:22.1091343Z * [new branch] gh/malfet/560/base -> origin/gh/malfet/560/base 2025-11-03T16:15:22.1091490Z * [new branch] gh/malfet/560/head -> origin/gh/malfet/560/head 2025-11-03T16:15:22.1091926Z * [new branch] gh/malfet/560/orig -> origin/gh/malfet/560/orig 2025-11-03T16:15:22.1096135Z * [new branch] gh/malfet/561/base -> origin/gh/malfet/561/base 2025-11-03T16:15:22.1096435Z * [new branch] gh/malfet/561/head -> origin/gh/malfet/561/head 2025-11-03T16:15:22.1096635Z * [new branch] gh/malfet/561/orig -> origin/gh/malfet/561/orig 2025-11-03T16:15:22.1096793Z * [new branch] gh/malfet/562/base -> origin/gh/malfet/562/base 2025-11-03T16:15:22.1097016Z * [new branch] gh/malfet/562/head -> origin/gh/malfet/562/head 2025-11-03T16:15:22.1097146Z * [new branch] gh/malfet/562/orig -> origin/gh/malfet/562/orig 2025-11-03T16:15:22.1097540Z * [new branch] gh/malfet/563/base -> origin/gh/malfet/563/base 2025-11-03T16:15:22.1098185Z * [new branch] gh/malfet/563/head -> origin/gh/malfet/563/head 2025-11-03T16:15:22.1098363Z * [new branch] gh/malfet/563/orig -> origin/gh/malfet/563/orig 2025-11-03T16:15:22.1102898Z * [new branch] gh/malfet/564/base -> origin/gh/malfet/564/base 2025-11-03T16:15:22.1103211Z * [new branch] gh/malfet/564/head -> origin/gh/malfet/564/head 2025-11-03T16:15:22.1103422Z * [new branch] gh/malfet/564/orig -> origin/gh/malfet/564/orig 2025-11-03T16:15:22.1103576Z * [new branch] gh/malfet/565/base -> origin/gh/malfet/565/base 2025-11-03T16:15:22.1103796Z * [new branch] gh/malfet/565/head -> origin/gh/malfet/565/head 2025-11-03T16:15:22.1103937Z * [new branch] gh/malfet/565/orig -> origin/gh/malfet/565/orig 2025-11-03T16:15:22.1104149Z * [new branch] gh/malfet/566/base -> origin/gh/malfet/566/base 2025-11-03T16:15:22.1104771Z * [new branch] gh/malfet/566/head -> origin/gh/malfet/566/head 2025-11-03T16:15:22.1104928Z * [new branch] gh/malfet/566/orig -> origin/gh/malfet/566/orig 2025-11-03T16:15:22.1106394Z * [new branch] gh/malfet/567/base -> origin/gh/malfet/567/base 2025-11-03T16:15:22.1106699Z * [new branch] gh/malfet/567/head -> origin/gh/malfet/567/head 2025-11-03T16:15:22.1106914Z * [new branch] gh/malfet/567/orig -> origin/gh/malfet/567/orig 2025-11-03T16:15:22.1108567Z * [new branch] gh/malfet/568/base -> origin/gh/malfet/568/base 2025-11-03T16:15:22.1108738Z * [new branch] gh/malfet/568/head -> origin/gh/malfet/568/head 2025-11-03T16:15:22.1108930Z * [new branch] gh/malfet/568/orig -> origin/gh/malfet/568/orig 2025-11-03T16:15:22.1110442Z * [new branch] gh/malfet/569/base -> origin/gh/malfet/569/base 2025-11-03T16:15:22.1110595Z * [new branch] gh/malfet/569/head -> origin/gh/malfet/569/head 2025-11-03T16:15:22.1111170Z * [new branch] gh/malfet/569/orig -> origin/gh/malfet/569/orig 2025-11-03T16:15:22.1112170Z * [new branch] gh/malfet/570/base -> origin/gh/malfet/570/base 2025-11-03T16:15:22.1112403Z * [new branch] gh/malfet/570/head -> origin/gh/malfet/570/head 2025-11-03T16:15:22.1113575Z * [new branch] gh/malfet/570/orig -> origin/gh/malfet/570/orig 2025-11-03T16:15:22.1114683Z * [new branch] gh/malfet/571/base -> origin/gh/malfet/571/base 2025-11-03T16:15:22.1115040Z * [new branch] gh/malfet/571/head -> origin/gh/malfet/571/head 2025-11-03T16:15:22.1118331Z * [new branch] gh/malfet/571/orig -> origin/gh/malfet/571/orig 2025-11-03T16:15:22.1118503Z * [new branch] gh/malfet/572/base -> origin/gh/malfet/572/base 2025-11-03T16:15:22.1118628Z * [new branch] gh/malfet/572/head -> origin/gh/malfet/572/head 2025-11-03T16:15:22.1118753Z * [new branch] gh/malfet/572/orig -> origin/gh/malfet/572/orig 2025-11-03T16:15:22.1124233Z * [new branch] gh/malfet/573/base -> origin/gh/malfet/573/base 2025-11-03T16:15:22.1124555Z * [new branch] gh/malfet/573/head -> origin/gh/malfet/573/head 2025-11-03T16:15:22.1124766Z * [new branch] gh/malfet/573/orig -> origin/gh/malfet/573/orig 2025-11-03T16:15:22.1124952Z * [new branch] gh/malfet/574/base -> origin/gh/malfet/574/base 2025-11-03T16:15:22.1125196Z * [new branch] gh/malfet/574/head -> origin/gh/malfet/574/head 2025-11-03T16:15:22.1125347Z * [new branch] gh/malfet/574/orig -> origin/gh/malfet/574/orig 2025-11-03T16:15:22.1125761Z * [new branch] gh/malfet/575/base -> origin/gh/malfet/575/base 2025-11-03T16:15:22.1125919Z * [new branch] gh/malfet/575/head -> origin/gh/malfet/575/head 2025-11-03T16:15:22.1126052Z * [new branch] gh/malfet/575/orig -> origin/gh/malfet/575/orig 2025-11-03T16:15:22.1126178Z * [new branch] gh/malfet/576/base -> origin/gh/malfet/576/base 2025-11-03T16:15:22.1126598Z * [new branch] gh/malfet/576/head -> origin/gh/malfet/576/head 2025-11-03T16:15:22.1127360Z * [new branch] gh/malfet/576/orig -> origin/gh/malfet/576/orig 2025-11-03T16:15:22.1131681Z * [new branch] gh/malfet/577/base -> origin/gh/malfet/577/base 2025-11-03T16:15:22.1132009Z * [new branch] gh/malfet/577/head -> origin/gh/malfet/577/head 2025-11-03T16:15:22.1132245Z * [new branch] gh/malfet/577/orig -> origin/gh/malfet/577/orig 2025-11-03T16:15:22.1132391Z * [new branch] gh/malfet/578/base -> origin/gh/malfet/578/base 2025-11-03T16:15:22.1132514Z * [new branch] gh/malfet/578/head -> origin/gh/malfet/578/head 2025-11-03T16:15:22.1132772Z * [new branch] gh/malfet/578/orig -> origin/gh/malfet/578/orig 2025-11-03T16:15:22.1133436Z * [new branch] gh/malfet/579/base -> origin/gh/malfet/579/base 2025-11-03T16:15:22.1133595Z * [new branch] gh/malfet/579/head -> origin/gh/malfet/579/head 2025-11-03T16:15:22.1133762Z * [new branch] gh/malfet/579/orig -> origin/gh/malfet/579/orig 2025-11-03T16:15:22.1137546Z * [new branch] gh/malfet/580/base -> origin/gh/malfet/580/base 2025-11-03T16:15:22.1137891Z * [new branch] gh/malfet/580/head -> origin/gh/malfet/580/head 2025-11-03T16:15:22.1138122Z * [new branch] gh/malfet/580/orig -> origin/gh/malfet/580/orig 2025-11-03T16:15:22.1138269Z * [new branch] gh/malfet/581/base -> origin/gh/malfet/581/base 2025-11-03T16:15:22.1138492Z * [new branch] gh/malfet/581/head -> origin/gh/malfet/581/head 2025-11-03T16:15:22.1138619Z * [new branch] gh/malfet/581/orig -> origin/gh/malfet/581/orig 2025-11-03T16:15:22.1139207Z * [new branch] gh/malfet/582/base -> origin/gh/malfet/582/base 2025-11-03T16:15:22.1139395Z * [new branch] gh/malfet/582/head -> origin/gh/malfet/582/head 2025-11-03T16:15:22.1140884Z * [new branch] gh/malfet/582/orig -> origin/gh/malfet/582/orig 2025-11-03T16:15:22.1141085Z * [new branch] gh/malfet/583/base -> origin/gh/malfet/583/base 2025-11-03T16:15:22.1141600Z * [new branch] gh/malfet/583/head -> origin/gh/malfet/583/head 2025-11-03T16:15:22.1143278Z * [new branch] gh/malfet/583/orig -> origin/gh/malfet/583/orig 2025-11-03T16:15:22.1143595Z * [new branch] gh/malfet/584/base -> origin/gh/malfet/584/base 2025-11-03T16:15:22.1143806Z * [new branch] gh/malfet/584/head -> origin/gh/malfet/584/head 2025-11-03T16:15:22.1144305Z * [new branch] gh/malfet/584/orig -> origin/gh/malfet/584/orig 2025-11-03T16:15:22.1145746Z * [new branch] gh/malfet/585/base -> origin/gh/malfet/585/base 2025-11-03T16:15:22.1146057Z * [new branch] gh/malfet/585/head -> origin/gh/malfet/585/head 2025-11-03T16:15:22.1146346Z * [new branch] gh/malfet/585/orig -> origin/gh/malfet/585/orig 2025-11-03T16:15:22.1147898Z * [new branch] gh/malfet/586/base -> origin/gh/malfet/586/base 2025-11-03T16:15:22.1148209Z * [new branch] gh/malfet/586/head -> origin/gh/malfet/586/head 2025-11-03T16:15:22.1148507Z * [new branch] gh/malfet/586/orig -> origin/gh/malfet/586/orig 2025-11-03T16:15:22.1150281Z * [new branch] gh/malfet/587/base -> origin/gh/malfet/587/base 2025-11-03T16:15:22.1150435Z * [new branch] gh/malfet/587/head -> origin/gh/malfet/587/head 2025-11-03T16:15:22.1150829Z * [new branch] gh/malfet/587/orig -> origin/gh/malfet/587/orig 2025-11-03T16:15:22.1152510Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-11-03T16:15:22.1152674Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-11-03T16:15:22.1153383Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-11-03T16:15:22.1154081Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-11-03T16:15:22.1154607Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-11-03T16:15:22.1157874Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-11-03T16:15:22.1160314Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-11-03T16:15:22.1160627Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-11-03T16:15:22.1160852Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-11-03T16:15:22.1161002Z * [new branch] gh/masnesral/236/base -> origin/gh/masnesral/236/base 2025-11-03T16:15:22.1161131Z * [new branch] gh/masnesral/236/head -> origin/gh/masnesral/236/head 2025-11-03T16:15:22.1161621Z * [new branch] gh/masnesral/236/orig -> origin/gh/masnesral/236/orig 2025-11-03T16:15:22.1166482Z * [new branch] gh/masnesral/237/base -> origin/gh/masnesral/237/base 2025-11-03T16:15:22.1166651Z * [new branch] gh/masnesral/237/head -> origin/gh/masnesral/237/head 2025-11-03T16:15:22.1166797Z * [new branch] gh/masnesral/237/orig -> origin/gh/masnesral/237/orig 2025-11-03T16:15:22.1166930Z * [new branch] gh/masnesral/238/base -> origin/gh/masnesral/238/base 2025-11-03T16:15:22.1167058Z * [new branch] gh/masnesral/238/head -> origin/gh/masnesral/238/head 2025-11-03T16:15:22.1167356Z * [new branch] gh/masnesral/238/orig -> origin/gh/masnesral/238/orig 2025-11-03T16:15:22.1167522Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-11-03T16:15:22.1167752Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-11-03T16:15:22.1167904Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-11-03T16:15:22.1168030Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-11-03T16:15:22.1172601Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-11-03T16:15:22.1172920Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-11-03T16:15:22.1173149Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-11-03T16:15:22.1173293Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-11-03T16:15:22.1173514Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-11-03T16:15:22.1173657Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-11-03T16:15:22.1173879Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-11-03T16:15:22.1174531Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-11-03T16:15:22.1174692Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-11-03T16:15:22.1174964Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-11-03T16:15:22.1180471Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-11-03T16:15:22.1180817Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-11-03T16:15:22.1181096Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-11-03T16:15:22.1181333Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-11-03T16:15:22.1181937Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-11-03T16:15:22.1182148Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-11-03T16:15:22.1182306Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-11-03T16:15:22.1182474Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-11-03T16:15:22.1182626Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-11-03T16:15:22.1182786Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-11-03T16:15:22.1182966Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-11-03T16:15:22.1183129Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-11-03T16:15:22.1188043Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-11-03T16:15:22.1188531Z * [new branch] gh/mikaylagawarecki/337/base -> origin/gh/mikaylagawarecki/337/base 2025-11-03T16:15:22.1188841Z * [new branch] gh/mikaylagawarecki/337/head -> origin/gh/mikaylagawarecki/337/head 2025-11-03T16:15:22.1189076Z * [new branch] gh/mikaylagawarecki/337/orig -> origin/gh/mikaylagawarecki/337/orig 2025-11-03T16:15:22.1189719Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-11-03T16:15:22.1189911Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-11-03T16:15:22.1190077Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-11-03T16:15:22.1190229Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-11-03T16:15:22.1190392Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-11-03T16:15:22.1190559Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-11-03T16:15:22.1190726Z * [new branch] gh/mikaylagawarecki/343/base -> origin/gh/mikaylagawarecki/343/base 2025-11-03T16:15:22.1190942Z * [new branch] gh/mikaylagawarecki/343/head -> origin/gh/mikaylagawarecki/343/head 2025-11-03T16:15:22.1193329Z * [new branch] gh/mikaylagawarecki/343/orig -> origin/gh/mikaylagawarecki/343/orig 2025-11-03T16:15:22.1193516Z * [new branch] gh/mikaylagawarecki/344/base -> origin/gh/mikaylagawarecki/344/base 2025-11-03T16:15:22.1193680Z * [new branch] gh/mikaylagawarecki/344/head -> origin/gh/mikaylagawarecki/344/head 2025-11-03T16:15:22.1193872Z * [new branch] gh/mikaylagawarecki/344/orig -> origin/gh/mikaylagawarecki/344/orig 2025-11-03T16:15:22.1195375Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-11-03T16:15:22.1195830Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-11-03T16:15:22.1197790Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-11-03T16:15:22.1198137Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-11-03T16:15:22.1198583Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-11-03T16:15:22.1198827Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-11-03T16:15:22.1202718Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-11-03T16:15:22.1203069Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-11-03T16:15:22.1203336Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-11-03T16:15:22.1203573Z * [new branch] gh/mikaylagawarecki/348/base -> origin/gh/mikaylagawarecki/348/base 2025-11-03T16:15:22.1203755Z * [new branch] gh/mikaylagawarecki/348/head -> origin/gh/mikaylagawarecki/348/head 2025-11-03T16:15:22.1204030Z * [new branch] gh/mikaylagawarecki/348/orig -> origin/gh/mikaylagawarecki/348/orig 2025-11-03T16:15:22.1204264Z * [new branch] gh/mikaylagawarecki/349/base -> origin/gh/mikaylagawarecki/349/base 2025-11-03T16:15:22.1204436Z * [new branch] gh/mikaylagawarecki/349/head -> origin/gh/mikaylagawarecki/349/head 2025-11-03T16:15:22.1205546Z * [new branch] gh/mikaylagawarecki/349/orig -> origin/gh/mikaylagawarecki/349/orig 2025-11-03T16:15:22.1207790Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-11-03T16:15:22.1208132Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-11-03T16:15:22.1208396Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-11-03T16:15:22.1209825Z * [new branch] gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base 2025-11-03T16:15:22.1210183Z * [new branch] gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head 2025-11-03T16:15:22.1210376Z * [new branch] gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig 2025-11-03T16:15:22.1214451Z * [new branch] gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base 2025-11-03T16:15:22.1214799Z * [new branch] gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head 2025-11-03T16:15:22.1215065Z * [new branch] gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig 2025-11-03T16:15:22.1215235Z * [new branch] gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base 2025-11-03T16:15:22.1215506Z * [new branch] gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head 2025-11-03T16:15:22.1215912Z * [new branch] gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig 2025-11-03T16:15:22.1217251Z * [new branch] gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base 2025-11-03T16:15:22.1217559Z * [new branch] gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head 2025-11-03T16:15:22.1217915Z * [new branch] gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig 2025-11-03T16:15:22.1219689Z * [new branch] gh/mikaylagawarecki/355/base -> origin/gh/mikaylagawarecki/355/base 2025-11-03T16:15:22.1220031Z * [new branch] gh/mikaylagawarecki/355/head -> origin/gh/mikaylagawarecki/355/head 2025-11-03T16:15:22.1220298Z * [new branch] gh/mikaylagawarecki/355/orig -> origin/gh/mikaylagawarecki/355/orig 2025-11-03T16:15:22.1221404Z * [new branch] gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base 2025-11-03T16:15:22.1221678Z * [new branch] gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head 2025-11-03T16:15:22.1223650Z * [new branch] gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig 2025-11-03T16:15:22.1224213Z * [new branch] gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base 2025-11-03T16:15:22.1224485Z * [new branch] gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head 2025-11-03T16:15:22.1224664Z * [new branch] gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig 2025-11-03T16:15:22.1225509Z * [new branch] gh/mikaylagawarecki/358/base -> origin/gh/mikaylagawarecki/358/base 2025-11-03T16:15:22.1225816Z * [new branch] gh/mikaylagawarecki/358/head -> origin/gh/mikaylagawarecki/358/head 2025-11-03T16:15:22.1228294Z * [new branch] gh/mikaylagawarecki/358/orig -> origin/gh/mikaylagawarecki/358/orig 2025-11-03T16:15:22.1228650Z * [new branch] gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base 2025-11-03T16:15:22.1228917Z * [new branch] gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head 2025-11-03T16:15:22.1229162Z * [new branch] gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig 2025-11-03T16:15:22.1229447Z * [new branch] gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base 2025-11-03T16:15:22.1230403Z * [new branch] gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head 2025-11-03T16:15:22.1230644Z * [new branch] gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig 2025-11-03T16:15:22.1232962Z * [new branch] gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base 2025-11-03T16:15:22.1233139Z * [new branch] gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head 2025-11-03T16:15:22.1233507Z * [new branch] gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig 2025-11-03T16:15:22.1234068Z * [new branch] gh/mlazos/20/base -> origin/gh/mlazos/20/base 2025-11-03T16:15:22.1239756Z * [new branch] gh/mlazos/20/head -> origin/gh/mlazos/20/head 2025-11-03T16:15:22.1241729Z * [new branch] gh/mlazos/20/orig -> origin/gh/mlazos/20/orig 2025-11-03T16:15:22.1242063Z * [new branch] gh/mlazos/21/base -> origin/gh/mlazos/21/base 2025-11-03T16:15:22.1242221Z * [new branch] gh/mlazos/21/head -> origin/gh/mlazos/21/head 2025-11-03T16:15:22.1242362Z * [new branch] gh/mlazos/21/orig -> origin/gh/mlazos/21/orig 2025-11-03T16:15:22.1242575Z * [new branch] gh/mlazos/22/base -> origin/gh/mlazos/22/base 2025-11-03T16:15:22.1242718Z * [new branch] gh/mlazos/22/head -> origin/gh/mlazos/22/head 2025-11-03T16:15:22.1242938Z * [new branch] gh/mlazos/22/orig -> origin/gh/mlazos/22/orig 2025-11-03T16:15:22.1243087Z * [new branch] gh/mlazos/23/base -> origin/gh/mlazos/23/base 2025-11-03T16:15:22.1243215Z * [new branch] gh/mlazos/23/head -> origin/gh/mlazos/23/head 2025-11-03T16:15:22.1243486Z * [new branch] gh/mlazos/23/orig -> origin/gh/mlazos/23/orig 2025-11-03T16:15:22.1247723Z * [new branch] gh/mlazos/24/base -> origin/gh/mlazos/24/base 2025-11-03T16:15:22.1248057Z * [new branch] gh/mlazos/24/head -> origin/gh/mlazos/24/head 2025-11-03T16:15:22.1248287Z * [new branch] gh/mlazos/24/orig -> origin/gh/mlazos/24/orig 2025-11-03T16:15:22.1248438Z * [new branch] gh/mlazos/25/base -> origin/gh/mlazos/25/base 2025-11-03T16:15:22.1248565Z * [new branch] gh/mlazos/25/head -> origin/gh/mlazos/25/head 2025-11-03T16:15:22.1253484Z * [new branch] gh/mlazos/25/orig -> origin/gh/mlazos/25/orig 2025-11-03T16:15:22.1253810Z * [new branch] gh/mlazos/26/base -> origin/gh/mlazos/26/base 2025-11-03T16:15:22.1254035Z * [new branch] gh/mlazos/26/head -> origin/gh/mlazos/26/head 2025-11-03T16:15:22.1254456Z * [new branch] gh/mlazos/26/orig -> origin/gh/mlazos/26/orig 2025-11-03T16:15:22.1255118Z * [new branch] gh/mlazos/27/base -> origin/gh/mlazos/27/base 2025-11-03T16:15:22.1255284Z * [new branch] gh/mlazos/27/head -> origin/gh/mlazos/27/head 2025-11-03T16:15:22.1255412Z * [new branch] gh/mlazos/27/orig -> origin/gh/mlazos/27/orig 2025-11-03T16:15:22.1255539Z * [new branch] gh/mlazos/28/base -> origin/gh/mlazos/28/base 2025-11-03T16:15:22.1255658Z * [new branch] gh/mlazos/28/head -> origin/gh/mlazos/28/head 2025-11-03T16:15:22.1255793Z * [new branch] gh/mlazos/28/orig -> origin/gh/mlazos/28/orig 2025-11-03T16:15:22.1255925Z * [new branch] gh/mlazos/29/base -> origin/gh/mlazos/29/base 2025-11-03T16:15:22.1260176Z * [new branch] gh/mlazos/29/head -> origin/gh/mlazos/29/head 2025-11-03T16:15:22.1260495Z * [new branch] gh/mlazos/29/orig -> origin/gh/mlazos/29/orig 2025-11-03T16:15:22.1260698Z * [new branch] gh/mlazos/30/base -> origin/gh/mlazos/30/base 2025-11-03T16:15:22.1260840Z * [new branch] gh/mlazos/30/head -> origin/gh/mlazos/30/head 2025-11-03T16:15:22.1261043Z * [new branch] gh/mlazos/30/orig -> origin/gh/mlazos/30/orig 2025-11-03T16:15:22.1261184Z * [new branch] gh/mlazos/31/base -> origin/gh/mlazos/31/base 2025-11-03T16:15:22.1261378Z * [new branch] gh/mlazos/31/head -> origin/gh/mlazos/31/head 2025-11-03T16:15:22.1262321Z * [new branch] gh/mlazos/31/orig -> origin/gh/mlazos/31/orig 2025-11-03T16:15:22.1262584Z * [new branch] gh/mlazos/32/base -> origin/gh/mlazos/32/base 2025-11-03T16:15:22.1262729Z * [new branch] gh/mlazos/32/head -> origin/gh/mlazos/32/head 2025-11-03T16:15:22.1262930Z * [new branch] gh/mlazos/32/orig -> origin/gh/mlazos/32/orig 2025-11-03T16:15:22.1263060Z * [new branch] gh/mlazos/33/base -> origin/gh/mlazos/33/base 2025-11-03T16:15:22.1263261Z * [new branch] gh/mlazos/33/head -> origin/gh/mlazos/33/head 2025-11-03T16:15:22.1263389Z * [new branch] gh/mlazos/33/orig -> origin/gh/mlazos/33/orig 2025-11-03T16:15:22.1268067Z * [new branch] gh/mlazos/34/base -> origin/gh/mlazos/34/base 2025-11-03T16:15:22.1268377Z * [new branch] gh/mlazos/34/head -> origin/gh/mlazos/34/head 2025-11-03T16:15:22.1268531Z * [new branch] gh/mlazos/34/orig -> origin/gh/mlazos/34/orig 2025-11-03T16:15:22.1268660Z * [new branch] gh/mlazos/35/base -> origin/gh/mlazos/35/base 2025-11-03T16:15:22.1268781Z * [new branch] gh/mlazos/35/head -> origin/gh/mlazos/35/head 2025-11-03T16:15:22.1269040Z * [new branch] gh/mlazos/35/orig -> origin/gh/mlazos/35/orig 2025-11-03T16:15:22.1269637Z * [new branch] gh/mlazos/36/base -> origin/gh/mlazos/36/base 2025-11-03T16:15:22.1269790Z * [new branch] gh/mlazos/36/head -> origin/gh/mlazos/36/head 2025-11-03T16:15:22.1269919Z * [new branch] gh/mlazos/36/orig -> origin/gh/mlazos/36/orig 2025-11-03T16:15:22.1270038Z * [new branch] gh/mlazos/37/base -> origin/gh/mlazos/37/base 2025-11-03T16:15:22.1270159Z * [new branch] gh/mlazos/37/head -> origin/gh/mlazos/37/head 2025-11-03T16:15:22.1270455Z * [new branch] gh/mlazos/37/orig -> origin/gh/mlazos/37/orig 2025-11-03T16:15:22.1274591Z * [new branch] gh/mlazos/38/base -> origin/gh/mlazos/38/base 2025-11-03T16:15:22.1274894Z * [new branch] gh/mlazos/38/head -> origin/gh/mlazos/38/head 2025-11-03T16:15:22.1275295Z * [new branch] gh/mlazos/38/orig -> origin/gh/mlazos/38/orig 2025-11-03T16:15:22.1275447Z * [new branch] gh/mlazos/39/base -> origin/gh/mlazos/39/base 2025-11-03T16:15:22.1275571Z * [new branch] gh/mlazos/39/head -> origin/gh/mlazos/39/head 2025-11-03T16:15:22.1275820Z * [new branch] gh/mlazos/39/orig -> origin/gh/mlazos/39/orig 2025-11-03T16:15:22.1275945Z * [new branch] gh/mlazos/40/base -> origin/gh/mlazos/40/base 2025-11-03T16:15:22.1276339Z * [new branch] gh/mlazos/40/head -> origin/gh/mlazos/40/head 2025-11-03T16:15:22.1278139Z * [new branch] gh/mlazos/40/orig -> origin/gh/mlazos/40/orig 2025-11-03T16:15:22.1278455Z * [new branch] gh/mlazos/41/base -> origin/gh/mlazos/41/base 2025-11-03T16:15:22.1278610Z * [new branch] gh/mlazos/41/head -> origin/gh/mlazos/41/head 2025-11-03T16:15:22.1280550Z * [new branch] gh/mlazos/41/orig -> origin/gh/mlazos/41/orig 2025-11-03T16:15:22.1280860Z * [new branch] gh/mlazos/42/base -> origin/gh/mlazos/42/base 2025-11-03T16:15:22.1281082Z * [new branch] gh/mlazos/42/head -> origin/gh/mlazos/42/head 2025-11-03T16:15:22.1282411Z * [new branch] gh/mlazos/42/orig -> origin/gh/mlazos/42/orig 2025-11-03T16:15:22.1282594Z * [new branch] gh/mlazos/43/base -> origin/gh/mlazos/43/base 2025-11-03T16:15:22.1285485Z * [new branch] gh/mlazos/43/head -> origin/gh/mlazos/43/head 2025-11-03T16:15:22.1285940Z * [new branch] gh/mlazos/43/orig -> origin/gh/mlazos/43/orig 2025-11-03T16:15:22.1286206Z * [new branch] gh/mlazos/44/base -> origin/gh/mlazos/44/base 2025-11-03T16:15:22.1286452Z * [new branch] gh/mlazos/44/head -> origin/gh/mlazos/44/head 2025-11-03T16:15:22.1286603Z * [new branch] gh/mlazos/44/orig -> origin/gh/mlazos/44/orig 2025-11-03T16:15:22.1287239Z * [new branch] gh/mlazos/45/base -> origin/gh/mlazos/45/base 2025-11-03T16:15:22.1287425Z * [new branch] gh/mlazos/45/head -> origin/gh/mlazos/45/head 2025-11-03T16:15:22.1288897Z * [new branch] gh/mlazos/45/orig -> origin/gh/mlazos/45/orig 2025-11-03T16:15:22.1289198Z * [new branch] gh/mlazos/46/base -> origin/gh/mlazos/46/base 2025-11-03T16:15:22.1291087Z * [new branch] gh/mlazos/46/head -> origin/gh/mlazos/46/head 2025-11-03T16:15:22.1291410Z * [new branch] gh/mlazos/46/orig -> origin/gh/mlazos/46/orig 2025-11-03T16:15:22.1291624Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-11-03T16:15:22.1293468Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-11-03T16:15:22.1293641Z * [new branch] gh/muchulee8/68/base -> origin/gh/muchulee8/68/base 2025-11-03T16:15:22.1293889Z * [new branch] gh/muchulee8/68/head -> origin/gh/muchulee8/68/head 2025-11-03T16:15:22.1295514Z * [new branch] gh/muchulee8/68/orig -> origin/gh/muchulee8/68/orig 2025-11-03T16:15:22.1295825Z * [new branch] gh/muchulee8/69/base -> origin/gh/muchulee8/69/base 2025-11-03T16:15:22.1296063Z * [new branch] gh/muchulee8/69/head -> origin/gh/muchulee8/69/head 2025-11-03T16:15:22.1297795Z * [new branch] gh/muchulee8/69/orig -> origin/gh/muchulee8/69/orig 2025-11-03T16:15:22.1300533Z * [new branch] gh/muchulee8/70/base -> origin/gh/muchulee8/70/base 2025-11-03T16:15:22.1300867Z * [new branch] gh/muchulee8/70/head -> origin/gh/muchulee8/70/head 2025-11-03T16:15:22.1301277Z * [new branch] gh/muchulee8/70/orig -> origin/gh/muchulee8/70/orig 2025-11-03T16:15:22.1301515Z * [new branch] gh/muchulee8/71/base -> origin/gh/muchulee8/71/base 2025-11-03T16:15:22.1301734Z * [new branch] gh/muchulee8/71/head -> origin/gh/muchulee8/71/head 2025-11-03T16:15:22.1301880Z * [new branch] gh/muchulee8/71/orig -> origin/gh/muchulee8/71/orig 2025-11-03T16:15:22.1302025Z * [new branch] gh/muchulee8/72/base -> origin/gh/muchulee8/72/base 2025-11-03T16:15:22.1302156Z * [new branch] gh/muchulee8/72/head -> origin/gh/muchulee8/72/head 2025-11-03T16:15:22.1302517Z * [new branch] gh/muchulee8/72/orig -> origin/gh/muchulee8/72/orig 2025-11-03T16:15:22.1304594Z * [new branch] gh/muchulee8/73/base -> origin/gh/muchulee8/73/base 2025-11-03T16:15:22.1304762Z * [new branch] gh/muchulee8/73/head -> origin/gh/muchulee8/73/head 2025-11-03T16:15:22.1304901Z * [new branch] gh/muchulee8/73/orig -> origin/gh/muchulee8/73/orig 2025-11-03T16:15:22.1306876Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-11-03T16:15:22.1307199Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-11-03T16:15:22.1307425Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-11-03T16:15:22.1309426Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-11-03T16:15:22.1309595Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-11-03T16:15:22.1309914Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-11-03T16:15:22.1310198Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-11-03T16:15:22.1311434Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-11-03T16:15:22.1311647Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-11-03T16:15:22.1314558Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-11-03T16:15:22.1318923Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-11-03T16:15:22.1323570Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-11-03T16:15:22.1325242Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-11-03T16:15:22.1325409Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-11-03T16:15:22.1325563Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-11-03T16:15:22.1325826Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-11-03T16:15:22.1331048Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-11-03T16:15:22.1332630Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-11-03T16:15:22.1332803Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-11-03T16:15:22.1332943Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-11-03T16:15:22.1333086Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-11-03T16:15:22.1333219Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-11-03T16:15:22.1333536Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-11-03T16:15:22.1333701Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-11-03T16:15:22.1334173Z * [new branch] gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base 2025-11-03T16:15:22.1334424Z * [new branch] gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head 2025-11-03T16:15:22.1334622Z * [new branch] gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig 2025-11-03T16:15:22.1334768Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-11-03T16:15:22.1334899Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-11-03T16:15:22.1335036Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-11-03T16:15:22.1335159Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-11-03T16:15:22.1335289Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-11-03T16:15:22.1335418Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-11-03T16:15:22.1335543Z * [new branch] gh/nikitaved/4/base -> origin/gh/nikitaved/4/base 2025-11-03T16:15:22.1335672Z * [new branch] gh/nikitaved/4/head -> origin/gh/nikitaved/4/head 2025-11-03T16:15:22.1335794Z * [new branch] gh/nikitaved/4/orig -> origin/gh/nikitaved/4/orig 2025-11-03T16:15:22.1335923Z * [new branch] gh/nikitaved/5/base -> origin/gh/nikitaved/5/base 2025-11-03T16:15:22.1336048Z * [new branch] gh/nikitaved/5/head -> origin/gh/nikitaved/5/head 2025-11-03T16:15:22.1336172Z * [new branch] gh/nikitaved/5/orig -> origin/gh/nikitaved/5/orig 2025-11-03T16:15:22.1336305Z * [new branch] gh/nikitaved/6/base -> origin/gh/nikitaved/6/base 2025-11-03T16:15:22.1336570Z * [new branch] gh/nikitaved/6/head -> origin/gh/nikitaved/6/head 2025-11-03T16:15:22.1336704Z * [new branch] gh/nikitaved/6/orig -> origin/gh/nikitaved/6/orig 2025-11-03T16:15:22.1336994Z * [new branch] gh/nikitaved/7/base -> origin/gh/nikitaved/7/base 2025-11-03T16:15:22.1338925Z * [new branch] gh/nikitaved/7/head -> origin/gh/nikitaved/7/head 2025-11-03T16:15:22.1339243Z * [new branch] gh/nikitaved/7/orig -> origin/gh/nikitaved/7/orig 2025-11-03T16:15:22.1339449Z * [new branch] gh/nikitaved/8/base -> origin/gh/nikitaved/8/base 2025-11-03T16:15:22.1339698Z * [new branch] gh/nikitaved/8/head -> origin/gh/nikitaved/8/head 2025-11-03T16:15:22.1341288Z * [new branch] gh/nikitaved/8/orig -> origin/gh/nikitaved/8/orig 2025-11-03T16:15:22.1341599Z * [new branch] gh/oulgen/1/base -> origin/gh/oulgen/1/base 2025-11-03T16:15:22.1341964Z * [new branch] gh/oulgen/1/head -> origin/gh/oulgen/1/head 2025-11-03T16:15:22.1343521Z * [new branch] gh/oulgen/1/orig -> origin/gh/oulgen/1/orig 2025-11-03T16:15:22.1343848Z * [new branch] gh/oulgen/2/base -> origin/gh/oulgen/2/base 2025-11-03T16:15:22.1344047Z * [new branch] gh/oulgen/2/head -> origin/gh/oulgen/2/head 2025-11-03T16:15:22.1346282Z * [new branch] gh/oulgen/2/orig -> origin/gh/oulgen/2/orig 2025-11-03T16:15:22.1346590Z * [new branch] gh/oulgen/3/base -> origin/gh/oulgen/3/base 2025-11-03T16:15:22.1346800Z * [new branch] gh/oulgen/3/head -> origin/gh/oulgen/3/head 2025-11-03T16:15:22.1346933Z * [new branch] gh/oulgen/3/orig -> origin/gh/oulgen/3/orig 2025-11-03T16:15:22.1347830Z * [new branch] gh/oulgen/4/base -> origin/gh/oulgen/4/base 2025-11-03T16:15:22.1348873Z * [new branch] gh/oulgen/4/head -> origin/gh/oulgen/4/head 2025-11-03T16:15:22.1349415Z * [new branch] gh/oulgen/4/orig -> origin/gh/oulgen/4/orig 2025-11-03T16:15:22.1349898Z * [new branch] gh/oulgen/5/base -> origin/gh/oulgen/5/base 2025-11-03T16:15:22.1350840Z * [new branch] gh/oulgen/5/head -> origin/gh/oulgen/5/head 2025-11-03T16:15:22.1351076Z * [new branch] gh/oulgen/5/orig -> origin/gh/oulgen/5/orig 2025-11-03T16:15:22.1353579Z * [new branch] gh/oulgen/6/base -> origin/gh/oulgen/6/base 2025-11-03T16:15:22.1353741Z * [new branch] gh/oulgen/6/head -> origin/gh/oulgen/6/head 2025-11-03T16:15:22.1353868Z * [new branch] gh/oulgen/6/orig -> origin/gh/oulgen/6/orig 2025-11-03T16:15:22.1354185Z * [new branch] gh/oulgen/7/base -> origin/gh/oulgen/7/base 2025-11-03T16:15:22.1356281Z * [new branch] gh/oulgen/7/head -> origin/gh/oulgen/7/head 2025-11-03T16:15:22.1356435Z * [new branch] gh/oulgen/7/orig -> origin/gh/oulgen/7/orig 2025-11-03T16:15:22.1356634Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-11-03T16:15:22.1358197Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-11-03T16:15:22.1358379Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-11-03T16:15:22.1361028Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-11-03T16:15:22.1361326Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-11-03T16:15:22.1361544Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-11-03T16:15:22.1361907Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-11-03T16:15:22.1362062Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-11-03T16:15:22.1362941Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-11-03T16:15:22.1363429Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-11-03T16:15:22.1366444Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-11-03T16:15:22.1366752Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-11-03T16:15:22.1366984Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-11-03T16:15:22.1367203Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-11-03T16:15:22.1367344Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-11-03T16:15:22.1367694Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-11-03T16:15:22.1371791Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-11-03T16:15:22.1372097Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-11-03T16:15:22.1372329Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-11-03T16:15:22.1372467Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-11-03T16:15:22.1372595Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-11-03T16:15:22.1372721Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-11-03T16:15:22.1372982Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-11-03T16:15:22.1373274Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-11-03T16:15:22.1374316Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-11-03T16:15:22.1374711Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-11-03T16:15:22.1376935Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-11-03T16:15:22.1377414Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-11-03T16:15:22.1377641Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-11-03T16:15:22.1377781Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-11-03T16:15:22.1378740Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-11-03T16:15:22.1381492Z * [new branch] gh/pearu/120/base -> origin/gh/pearu/120/base 2025-11-03T16:15:22.1381807Z * [new branch] gh/pearu/120/head -> origin/gh/pearu/120/head 2025-11-03T16:15:22.1381970Z * [new branch] gh/pearu/120/orig -> origin/gh/pearu/120/orig 2025-11-03T16:15:22.1382106Z * [new branch] gh/pearu/121/base -> origin/gh/pearu/121/base 2025-11-03T16:15:22.1382350Z * [new branch] gh/pearu/121/head -> origin/gh/pearu/121/head 2025-11-03T16:15:22.1383030Z * [new branch] gh/pearu/121/orig -> origin/gh/pearu/121/orig 2025-11-03T16:15:22.1383487Z * [new branch] gh/pearu/122/base -> origin/gh/pearu/122/base 2025-11-03T16:15:22.1385179Z * [new branch] gh/pearu/122/head -> origin/gh/pearu/122/head 2025-11-03T16:15:22.1385342Z * [new branch] gh/pearu/122/orig -> origin/gh/pearu/122/orig 2025-11-03T16:15:22.1387303Z * [new branch] gh/pearu/138/base -> origin/gh/pearu/138/base 2025-11-03T16:15:22.1387457Z * [new branch] gh/pearu/138/head -> origin/gh/pearu/138/head 2025-11-03T16:15:22.1387724Z * [new branch] gh/pearu/138/orig -> origin/gh/pearu/138/orig 2025-11-03T16:15:22.1389503Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-11-03T16:15:22.1389665Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-11-03T16:15:22.1389805Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-11-03T16:15:22.1390956Z * [new branch] gh/pearu/140/base -> origin/gh/pearu/140/base 2025-11-03T16:15:22.1391187Z * [new branch] gh/pearu/140/head -> origin/gh/pearu/140/head 2025-11-03T16:15:22.1392196Z * [new branch] gh/pearu/140/orig -> origin/gh/pearu/140/orig 2025-11-03T16:15:22.1393315Z * [new branch] gh/pearu/141/base -> origin/gh/pearu/141/base 2025-11-03T16:15:22.1393758Z * [new branch] gh/pearu/141/head -> origin/gh/pearu/141/head 2025-11-03T16:15:22.1395035Z * [new branch] gh/pearu/141/orig -> origin/gh/pearu/141/orig 2025-11-03T16:15:22.1395317Z * [new branch] gh/pearu/142/base -> origin/gh/pearu/142/base 2025-11-03T16:15:22.1397472Z * [new branch] gh/pearu/142/head -> origin/gh/pearu/142/head 2025-11-03T16:15:22.1397795Z * [new branch] gh/pearu/142/orig -> origin/gh/pearu/142/orig 2025-11-03T16:15:22.1397938Z * [new branch] gh/pearu/143/base -> origin/gh/pearu/143/base 2025-11-03T16:15:22.1398129Z * [new branch] gh/pearu/143/head -> origin/gh/pearu/143/head 2025-11-03T16:15:22.1398959Z * [new branch] gh/pearu/143/orig -> origin/gh/pearu/143/orig 2025-11-03T16:15:22.1401120Z * [new branch] gh/pearu/144/base -> origin/gh/pearu/144/base 2025-11-03T16:15:22.1401429Z * [new branch] gh/pearu/144/head -> origin/gh/pearu/144/head 2025-11-03T16:15:22.1401662Z * [new branch] gh/pearu/144/orig -> origin/gh/pearu/144/orig 2025-11-03T16:15:22.1401800Z * [new branch] gh/pearu/145/base -> origin/gh/pearu/145/base 2025-11-03T16:15:22.1403037Z * [new branch] gh/pearu/145/head -> origin/gh/pearu/145/head 2025-11-03T16:15:22.1403508Z * [new branch] gh/pearu/145/orig -> origin/gh/pearu/145/orig 2025-11-03T16:15:22.1403761Z * [new branch] gh/pearu/146/base -> origin/gh/pearu/146/base 2025-11-03T16:15:22.1405315Z * [new branch] gh/pearu/146/head -> origin/gh/pearu/146/head 2025-11-03T16:15:22.1405631Z * [new branch] gh/pearu/147/base -> origin/gh/pearu/147/base 2025-11-03T16:15:22.1405865Z * [new branch] gh/pearu/147/head -> origin/gh/pearu/147/head 2025-11-03T16:15:22.1407005Z * [new branch] gh/pearu/147/orig -> origin/gh/pearu/147/orig 2025-11-03T16:15:22.1407278Z * [new branch] gh/pearu/148/base -> origin/gh/pearu/148/base 2025-11-03T16:15:22.1408150Z * [new branch] gh/pearu/148/head -> origin/gh/pearu/148/head 2025-11-03T16:15:22.1408596Z * [new branch] gh/pearu/148/orig -> origin/gh/pearu/148/orig 2025-11-03T16:15:22.1410931Z * [new branch] gh/pearu/149/base -> origin/gh/pearu/149/base 2025-11-03T16:15:22.1411239Z * [new branch] gh/pearu/149/head -> origin/gh/pearu/149/head 2025-11-03T16:15:22.1411449Z * [new branch] gh/pearu/149/orig -> origin/gh/pearu/149/orig 2025-11-03T16:15:22.1411815Z * [new branch] gh/pearu/150/base -> origin/gh/pearu/150/base 2025-11-03T16:15:22.1412775Z * [new branch] gh/pearu/150/head -> origin/gh/pearu/150/head 2025-11-03T16:15:22.1413044Z * [new branch] gh/pearu/150/orig -> origin/gh/pearu/150/orig 2025-11-03T16:15:22.1417507Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-11-03T16:15:22.1417676Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-11-03T16:15:22.1417808Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-11-03T16:15:22.1417934Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-11-03T16:15:22.1418216Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-11-03T16:15:22.1419698Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-11-03T16:15:22.1419896Z * [new branch] gh/pianpwk/1/base -> origin/gh/pianpwk/1/base 2025-11-03T16:15:22.1422203Z * [new branch] gh/pianpwk/1/head -> origin/gh/pianpwk/1/head 2025-11-03T16:15:22.1422511Z * [new branch] gh/pianpwk/1/orig -> origin/gh/pianpwk/1/orig 2025-11-03T16:15:22.1422754Z * [new branch] gh/pianpwk/10/base -> origin/gh/pianpwk/10/base 2025-11-03T16:15:22.1422896Z * [new branch] gh/pianpwk/10/head -> origin/gh/pianpwk/10/head 2025-11-03T16:15:22.1423309Z * [new branch] gh/pianpwk/10/orig -> origin/gh/pianpwk/10/orig 2025-11-03T16:15:22.1427399Z * [new branch] gh/pianpwk/11/base -> origin/gh/pianpwk/11/base 2025-11-03T16:15:22.1427555Z * [new branch] gh/pianpwk/11/head -> origin/gh/pianpwk/11/head 2025-11-03T16:15:22.1427677Z * [new branch] gh/pianpwk/11/orig -> origin/gh/pianpwk/11/orig 2025-11-03T16:15:22.1427805Z * [new branch] gh/pianpwk/12/base -> origin/gh/pianpwk/12/base 2025-11-03T16:15:22.1427934Z * [new branch] gh/pianpwk/12/head -> origin/gh/pianpwk/12/head 2025-11-03T16:15:22.1428557Z * [new branch] gh/pianpwk/12/orig -> origin/gh/pianpwk/12/orig 2025-11-03T16:15:22.1432384Z * [new branch] gh/pianpwk/13/base -> origin/gh/pianpwk/13/base 2025-11-03T16:15:22.1432553Z * [new branch] gh/pianpwk/13/head -> origin/gh/pianpwk/13/head 2025-11-03T16:15:22.1432679Z * [new branch] gh/pianpwk/13/orig -> origin/gh/pianpwk/13/orig 2025-11-03T16:15:22.1433027Z * [new branch] gh/pianpwk/14/base -> origin/gh/pianpwk/14/base 2025-11-03T16:15:22.1433153Z * [new branch] gh/pianpwk/14/head -> origin/gh/pianpwk/14/head 2025-11-03T16:15:22.1433452Z * [new branch] gh/pianpwk/14/orig -> origin/gh/pianpwk/14/orig 2025-11-03T16:15:22.1433876Z * [new branch] gh/pianpwk/15/base -> origin/gh/pianpwk/15/base 2025-11-03T16:15:22.1434738Z * [new branch] gh/pianpwk/15/head -> origin/gh/pianpwk/15/head 2025-11-03T16:15:22.1435194Z * [new branch] gh/pianpwk/15/orig -> origin/gh/pianpwk/15/orig 2025-11-03T16:15:22.1437786Z * [new branch] gh/pianpwk/16/base -> origin/gh/pianpwk/16/base 2025-11-03T16:15:22.1441225Z * [new branch] gh/pianpwk/16/head -> origin/gh/pianpwk/16/head 2025-11-03T16:15:22.1441543Z * [new branch] gh/pianpwk/16/orig -> origin/gh/pianpwk/16/orig 2025-11-03T16:15:22.1441795Z * [new branch] gh/pianpwk/17/base -> origin/gh/pianpwk/17/base 2025-11-03T16:15:22.1441953Z * [new branch] gh/pianpwk/17/head -> origin/gh/pianpwk/17/head 2025-11-03T16:15:22.1442078Z * [new branch] gh/pianpwk/17/orig -> origin/gh/pianpwk/17/orig 2025-11-03T16:15:22.1442325Z * [new branch] gh/pianpwk/18/base -> origin/gh/pianpwk/18/base 2025-11-03T16:15:22.1444931Z * [new branch] gh/pianpwk/18/head -> origin/gh/pianpwk/18/head 2025-11-03T16:15:22.1445172Z * [new branch] gh/pianpwk/19/base -> origin/gh/pianpwk/19/base 2025-11-03T16:15:22.1445443Z * [new branch] gh/pianpwk/19/head -> origin/gh/pianpwk/19/head 2025-11-03T16:15:22.1445691Z * [new branch] gh/pianpwk/19/orig -> origin/gh/pianpwk/19/orig 2025-11-03T16:15:22.1445854Z * [new branch] gh/pianpwk/2/base -> origin/gh/pianpwk/2/base 2025-11-03T16:15:22.1446080Z * [new branch] gh/pianpwk/2/head -> origin/gh/pianpwk/2/head 2025-11-03T16:15:22.1446696Z * [new branch] gh/pianpwk/2/orig -> origin/gh/pianpwk/2/orig 2025-11-03T16:15:22.1452128Z * [new branch] gh/pianpwk/20/base -> origin/gh/pianpwk/20/base 2025-11-03T16:15:22.1452452Z * [new branch] gh/pianpwk/20/head -> origin/gh/pianpwk/20/head 2025-11-03T16:15:22.1452679Z * [new branch] gh/pianpwk/20/orig -> origin/gh/pianpwk/20/orig 2025-11-03T16:15:22.1452834Z * [new branch] gh/pianpwk/21/base -> origin/gh/pianpwk/21/base 2025-11-03T16:15:22.1452993Z * [new branch] gh/pianpwk/21/head -> origin/gh/pianpwk/21/head 2025-11-03T16:15:22.1453117Z * [new branch] gh/pianpwk/22/base -> origin/gh/pianpwk/22/base 2025-11-03T16:15:22.1453393Z * [new branch] gh/pianpwk/22/head -> origin/gh/pianpwk/22/head 2025-11-03T16:15:22.1453536Z * [new branch] gh/pianpwk/22/orig -> origin/gh/pianpwk/22/orig 2025-11-03T16:15:22.1453765Z * [new branch] gh/pianpwk/23/base -> origin/gh/pianpwk/23/base 2025-11-03T16:15:22.1454401Z * [new branch] gh/pianpwk/23/head -> origin/gh/pianpwk/23/head 2025-11-03T16:15:22.1454566Z * [new branch] gh/pianpwk/23/orig -> origin/gh/pianpwk/23/orig 2025-11-03T16:15:22.1454691Z * [new branch] gh/pianpwk/24/base -> origin/gh/pianpwk/24/base 2025-11-03T16:15:22.1454822Z * [new branch] gh/pianpwk/24/head -> origin/gh/pianpwk/24/head 2025-11-03T16:15:22.1458852Z * [new branch] gh/pianpwk/24/orig -> origin/gh/pianpwk/24/orig 2025-11-03T16:15:22.1459170Z * [new branch] gh/pianpwk/25/base -> origin/gh/pianpwk/25/base 2025-11-03T16:15:22.1459561Z * [new branch] gh/pianpwk/25/head -> origin/gh/pianpwk/25/head 2025-11-03T16:15:22.1459713Z * [new branch] gh/pianpwk/25/orig -> origin/gh/pianpwk/25/orig 2025-11-03T16:15:22.1459936Z * [new branch] gh/pianpwk/26/base -> origin/gh/pianpwk/26/base 2025-11-03T16:15:22.1460537Z * [new branch] gh/pianpwk/26/head -> origin/gh/pianpwk/26/head 2025-11-03T16:15:22.1460703Z * [new branch] gh/pianpwk/26/orig -> origin/gh/pianpwk/26/orig 2025-11-03T16:15:22.1460829Z * [new branch] gh/pianpwk/27/base -> origin/gh/pianpwk/27/base 2025-11-03T16:15:22.1460958Z * [new branch] gh/pianpwk/27/head -> origin/gh/pianpwk/27/head 2025-11-03T16:15:22.1461093Z * [new branch] gh/pianpwk/27/orig -> origin/gh/pianpwk/27/orig 2025-11-03T16:15:22.1465284Z * [new branch] gh/pianpwk/28/base -> origin/gh/pianpwk/28/base 2025-11-03T16:15:22.1465615Z * [new branch] gh/pianpwk/28/head -> origin/gh/pianpwk/28/head 2025-11-03T16:15:22.1465839Z * [new branch] gh/pianpwk/28/orig -> origin/gh/pianpwk/28/orig 2025-11-03T16:15:22.1465992Z * [new branch] gh/pianpwk/3/base -> origin/gh/pianpwk/3/base 2025-11-03T16:15:22.1466121Z * [new branch] gh/pianpwk/3/head -> origin/gh/pianpwk/3/head 2025-11-03T16:15:22.1466362Z * [new branch] gh/pianpwk/3/orig -> origin/gh/pianpwk/3/orig 2025-11-03T16:15:22.1466487Z * [new branch] gh/pianpwk/4/base -> origin/gh/pianpwk/4/base 2025-11-03T16:15:22.1469108Z * [new branch] gh/pianpwk/4/head -> origin/gh/pianpwk/4/head 2025-11-03T16:15:22.1469449Z * [new branch] gh/pianpwk/4/orig -> origin/gh/pianpwk/4/orig 2025-11-03T16:15:22.1469587Z * [new branch] gh/pianpwk/5/base -> origin/gh/pianpwk/5/base 2025-11-03T16:15:22.1469736Z * [new branch] gh/pianpwk/5/head -> origin/gh/pianpwk/5/head 2025-11-03T16:15:22.1469860Z * [new branch] gh/pianpwk/5/orig -> origin/gh/pianpwk/5/orig 2025-11-03T16:15:22.1469988Z * [new branch] gh/pianpwk/6/base -> origin/gh/pianpwk/6/base 2025-11-03T16:15:22.1470268Z * [new branch] gh/pianpwk/6/head -> origin/gh/pianpwk/6/head 2025-11-03T16:15:22.1470403Z * [new branch] gh/pianpwk/6/orig -> origin/gh/pianpwk/6/orig 2025-11-03T16:15:22.1471366Z * [new branch] gh/pianpwk/7/base -> origin/gh/pianpwk/7/base 2025-11-03T16:15:22.1471993Z * [new branch] gh/pianpwk/7/head -> origin/gh/pianpwk/7/head 2025-11-03T16:15:22.1472389Z * [new branch] gh/pianpwk/7/orig -> origin/gh/pianpwk/7/orig 2025-11-03T16:15:22.1473344Z * [new branch] gh/pianpwk/8/base -> origin/gh/pianpwk/8/base 2025-11-03T16:15:22.1473643Z * [new branch] gh/pianpwk/8/head -> origin/gh/pianpwk/8/head 2025-11-03T16:15:22.1474448Z * [new branch] gh/pianpwk/8/orig -> origin/gh/pianpwk/8/orig 2025-11-03T16:15:22.1477707Z * [new branch] gh/pianpwk/9/base -> origin/gh/pianpwk/9/base 2025-11-03T16:15:22.1478019Z * [new branch] gh/pianpwk/9/head -> origin/gh/pianpwk/9/head 2025-11-03T16:15:22.1478241Z * [new branch] gh/pianpwk/9/orig -> origin/gh/pianpwk/9/orig 2025-11-03T16:15:22.1478444Z * [new branch] gh/qqaatw/29/head -> origin/gh/qqaatw/29/head 2025-11-03T16:15:22.1479036Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-11-03T16:15:22.1481303Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-11-03T16:15:22.1481619Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-11-03T16:15:22.1482017Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-11-03T16:15:22.1482160Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-11-03T16:15:22.1483009Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-11-03T16:15:22.1483366Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-11-03T16:15:22.1486389Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-11-03T16:15:22.1486699Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-11-03T16:15:22.1486907Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-11-03T16:15:22.1487064Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-11-03T16:15:22.1487285Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-11-03T16:15:22.1488083Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-11-03T16:15:22.1488697Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-11-03T16:15:22.1491347Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-11-03T16:15:22.1491698Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-11-03T16:15:22.1491963Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-11-03T16:15:22.1492198Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-11-03T16:15:22.1492431Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-11-03T16:15:22.1492974Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-11-03T16:15:22.1497408Z * [new branch] gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base 2025-11-03T16:15:22.1497768Z * [new branch] gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head 2025-11-03T16:15:22.1498028Z * [new branch] gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig 2025-11-03T16:15:22.1498262Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-11-03T16:15:22.1498406Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-11-03T16:15:22.1499040Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-11-03T16:15:22.1499190Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-11-03T16:15:22.1499337Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-11-03T16:15:22.1499816Z * [new branch] gh/rtimpe/16/base -> origin/gh/rtimpe/16/base 2025-11-03T16:15:22.1500764Z * [new branch] gh/rtimpe/16/head -> origin/gh/rtimpe/16/head 2025-11-03T16:15:22.1501111Z * [new branch] gh/rtimpe/16/orig -> origin/gh/rtimpe/16/orig 2025-11-03T16:15:22.1503571Z * [new branch] gh/rtimpe/19/base -> origin/gh/rtimpe/19/base 2025-11-03T16:15:22.1503875Z * [new branch] gh/rtimpe/19/head -> origin/gh/rtimpe/19/head 2025-11-03T16:15:22.1504088Z * [new branch] gh/rtimpe/19/orig -> origin/gh/rtimpe/19/orig 2025-11-03T16:15:22.1504314Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-11-03T16:15:22.1504448Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-11-03T16:15:22.1508099Z * [new branch] gh/rtimpe/20/base -> origin/gh/rtimpe/20/base 2025-11-03T16:15:22.1508400Z * [new branch] gh/rtimpe/20/head -> origin/gh/rtimpe/20/head 2025-11-03T16:15:22.1508611Z * [new branch] gh/rtimpe/20/orig -> origin/gh/rtimpe/20/orig 2025-11-03T16:15:22.1508989Z * [new branch] gh/rtimpe/21/base -> origin/gh/rtimpe/21/base 2025-11-03T16:15:22.1509131Z * [new branch] gh/rtimpe/21/head -> origin/gh/rtimpe/21/head 2025-11-03T16:15:22.1509330Z * [new branch] gh/rtimpe/21/orig -> origin/gh/rtimpe/21/orig 2025-11-03T16:15:22.1509978Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-11-03T16:15:22.1510242Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-11-03T16:15:22.1511659Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-11-03T16:15:22.1511829Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-11-03T16:15:22.1514212Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-11-03T16:15:22.1514394Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-11-03T16:15:22.1519295Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-11-03T16:15:22.1519457Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-11-03T16:15:22.1519592Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-11-03T16:15:22.1519733Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-11-03T16:15:22.1520106Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-11-03T16:15:22.1521515Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-11-03T16:15:22.1521889Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-11-03T16:15:22.1522370Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-11-03T16:15:22.1523754Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-11-03T16:15:22.1523950Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-11-03T16:15:22.1526669Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-11-03T16:15:22.1526993Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-11-03T16:15:22.1527235Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-11-03T16:15:22.1527384Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-11-03T16:15:22.1527542Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-11-03T16:15:22.1528041Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-11-03T16:15:22.1531804Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-11-03T16:15:22.1532131Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-11-03T16:15:22.1532355Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-11-03T16:15:22.1532511Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-11-03T16:15:22.1532762Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-11-03T16:15:22.1532907Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-11-03T16:15:22.1533632Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-11-03T16:15:22.1534235Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-11-03T16:15:22.1534553Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-11-03T16:15:22.1536697Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-11-03T16:15:22.1537016Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-11-03T16:15:22.1537231Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-11-03T16:15:22.1537384Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-11-03T16:15:22.1538890Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-11-03T16:15:22.1539209Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-11-03T16:15:22.1539609Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-11-03T16:15:22.1541316Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-11-03T16:15:22.1541644Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-11-03T16:15:22.1541868Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-11-03T16:15:22.1542355Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-11-03T16:15:22.1543521Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-11-03T16:15:22.1543818Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-11-03T16:15:22.1545997Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-11-03T16:15:22.1546317Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-11-03T16:15:22.1546681Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-11-03T16:15:22.1546953Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-11-03T16:15:22.1547099Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-11-03T16:15:22.1548304Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-11-03T16:15:22.1548868Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-11-03T16:15:22.1549138Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-11-03T16:15:22.1550674Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-11-03T16:15:22.1550847Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-11-03T16:15:22.1551314Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-11-03T16:15:22.1552335Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-11-03T16:15:22.1552655Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-11-03T16:15:22.1554094Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-11-03T16:15:22.1554261Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-11-03T16:15:22.1555074Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-11-03T16:15:22.1555472Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-11-03T16:15:22.1558631Z * [new branch] gh/seemethere/72/base -> origin/gh/seemethere/72/base 2025-11-03T16:15:22.1558797Z * [new branch] gh/seemethere/72/head -> origin/gh/seemethere/72/head 2025-11-03T16:15:22.1558928Z * [new branch] gh/seemethere/72/orig -> origin/gh/seemethere/72/orig 2025-11-03T16:15:22.1559076Z * [new branch] gh/seemethere/73/base -> origin/gh/seemethere/73/base 2025-11-03T16:15:22.1559573Z * [new branch] gh/seemethere/73/head -> origin/gh/seemethere/73/head 2025-11-03T16:15:22.1560495Z * [new branch] gh/seemethere/73/orig -> origin/gh/seemethere/73/orig 2025-11-03T16:15:22.1563724Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-11-03T16:15:22.1564049Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-11-03T16:15:22.1564280Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-11-03T16:15:22.1564427Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-11-03T16:15:22.1565894Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-11-03T16:15:22.1566224Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-11-03T16:15:22.1566586Z * [new branch] gh/shunting314/217/base -> origin/gh/shunting314/217/base 2025-11-03T16:15:22.1568212Z * [new branch] gh/shunting314/217/head -> origin/gh/shunting314/217/head 2025-11-03T16:15:22.1568558Z * [new branch] gh/shunting314/217/orig -> origin/gh/shunting314/217/orig 2025-11-03T16:15:22.1568794Z * [new branch] gh/shunting314/228/base -> origin/gh/shunting314/228/base 2025-11-03T16:15:22.1571112Z * [new branch] gh/shunting314/228/head -> origin/gh/shunting314/228/head 2025-11-03T16:15:22.1571437Z * [new branch] gh/shunting314/228/orig -> origin/gh/shunting314/228/orig 2025-11-03T16:15:22.1571678Z * [new branch] gh/shunting314/229/base -> origin/gh/shunting314/229/base 2025-11-03T16:15:22.1571841Z * [new branch] gh/shunting314/229/head -> origin/gh/shunting314/229/head 2025-11-03T16:15:22.1572304Z * [new branch] gh/shunting314/229/orig -> origin/gh/shunting314/229/orig 2025-11-03T16:15:22.1575850Z * [new branch] gh/shunting314/230/base -> origin/gh/shunting314/230/base 2025-11-03T16:15:22.1576188Z * [new branch] gh/shunting314/230/head -> origin/gh/shunting314/230/head 2025-11-03T16:15:22.1576422Z * [new branch] gh/shunting314/230/orig -> origin/gh/shunting314/230/orig 2025-11-03T16:15:22.1576580Z * [new branch] gh/shunting314/231/base -> origin/gh/shunting314/231/base 2025-11-03T16:15:22.1576806Z * [new branch] gh/shunting314/231/head -> origin/gh/shunting314/231/head 2025-11-03T16:15:22.1576955Z * [new branch] gh/shunting314/231/orig -> origin/gh/shunting314/231/orig 2025-11-03T16:15:22.1578145Z * [new branch] gh/shunting314/232/base -> origin/gh/shunting314/232/base 2025-11-03T16:15:22.1578481Z * [new branch] gh/shunting314/232/head -> origin/gh/shunting314/232/head 2025-11-03T16:15:22.1581249Z * [new branch] gh/shunting314/232/orig -> origin/gh/shunting314/232/orig 2025-11-03T16:15:22.1581577Z * [new branch] gh/shunting314/233/base -> origin/gh/shunting314/233/base 2025-11-03T16:15:22.1581839Z * [new branch] gh/shunting314/233/head -> origin/gh/shunting314/233/head 2025-11-03T16:15:22.1581996Z * [new branch] gh/shunting314/233/orig -> origin/gh/shunting314/233/orig 2025-11-03T16:15:22.1582396Z * [new branch] gh/shunting314/234/base -> origin/gh/shunting314/234/base 2025-11-03T16:15:22.1586753Z * [new branch] gh/shunting314/234/head -> origin/gh/shunting314/234/head 2025-11-03T16:15:22.1587082Z * [new branch] gh/shunting314/234/orig -> origin/gh/shunting314/234/orig 2025-11-03T16:15:22.1587317Z * [new branch] gh/shunting314/235/base -> origin/gh/shunting314/235/base 2025-11-03T16:15:22.1587492Z * [new branch] gh/shunting314/235/head -> origin/gh/shunting314/235/head 2025-11-03T16:15:22.1587632Z * [new branch] gh/shunting314/235/orig -> origin/gh/shunting314/235/orig 2025-11-03T16:15:22.1588048Z * [new branch] gh/shunting314/236/base -> origin/gh/shunting314/236/base 2025-11-03T16:15:22.1588206Z * [new branch] gh/shunting314/236/head -> origin/gh/shunting314/236/head 2025-11-03T16:15:22.1588341Z * [new branch] gh/shunting314/236/orig -> origin/gh/shunting314/236/orig 2025-11-03T16:15:22.1590147Z * [new branch] gh/shunting314/237/base -> origin/gh/shunting314/237/base 2025-11-03T16:15:22.1590310Z * [new branch] gh/shunting314/237/head -> origin/gh/shunting314/237/head 2025-11-03T16:15:22.1590900Z * [new branch] gh/shunting314/237/orig -> origin/gh/shunting314/237/orig 2025-11-03T16:15:22.1595122Z * [new branch] gh/shunting314/238/base -> origin/gh/shunting314/238/base 2025-11-03T16:15:22.1595432Z * [new branch] gh/shunting314/238/head -> origin/gh/shunting314/238/head 2025-11-03T16:15:22.1595599Z * [new branch] gh/shunting314/238/orig -> origin/gh/shunting314/238/orig 2025-11-03T16:15:22.1595736Z * [new branch] gh/shunting314/239/base -> origin/gh/shunting314/239/base 2025-11-03T16:15:22.1595998Z * [new branch] gh/shunting314/239/head -> origin/gh/shunting314/239/head 2025-11-03T16:15:22.1596142Z * [new branch] gh/shunting314/239/orig -> origin/gh/shunting314/239/orig 2025-11-03T16:15:22.1596402Z * [new branch] gh/shunting314/240/base -> origin/gh/shunting314/240/base 2025-11-03T16:15:22.1598027Z * [new branch] gh/shunting314/240/head -> origin/gh/shunting314/240/head 2025-11-03T16:15:22.1598350Z * [new branch] gh/shunting314/240/orig -> origin/gh/shunting314/240/orig 2025-11-03T16:15:22.1598890Z * [new branch] gh/shunting314/241/base -> origin/gh/shunting314/241/base 2025-11-03T16:15:22.1600265Z * [new branch] gh/shunting314/241/head -> origin/gh/shunting314/241/head 2025-11-03T16:15:22.1600601Z * [new branch] gh/shunting314/241/orig -> origin/gh/shunting314/241/orig 2025-11-03T16:15:22.1600979Z * [new branch] gh/shunting314/242/base -> origin/gh/shunting314/242/base 2025-11-03T16:15:22.1603116Z * [new branch] gh/shunting314/242/head -> origin/gh/shunting314/242/head 2025-11-03T16:15:22.1603438Z * [new branch] gh/shunting314/242/orig -> origin/gh/shunting314/242/orig 2025-11-03T16:15:22.1603666Z * [new branch] gh/shunting314/243/base -> origin/gh/shunting314/243/base 2025-11-03T16:15:22.1603811Z * [new branch] gh/shunting314/243/head -> origin/gh/shunting314/243/head 2025-11-03T16:15:22.1604764Z * [new branch] gh/shunting314/243/orig -> origin/gh/shunting314/243/orig 2025-11-03T16:15:22.1608240Z * [new branch] gh/shunting314/244/base -> origin/gh/shunting314/244/base 2025-11-03T16:15:22.1608565Z * [new branch] gh/shunting314/244/head -> origin/gh/shunting314/244/head 2025-11-03T16:15:22.1608806Z * [new branch] gh/shunting314/244/orig -> origin/gh/shunting314/244/orig 2025-11-03T16:15:22.1608960Z * [new branch] gh/shunting314/245/base -> origin/gh/shunting314/245/base 2025-11-03T16:15:22.1609196Z * [new branch] gh/shunting314/245/head -> origin/gh/shunting314/245/head 2025-11-03T16:15:22.1609349Z * [new branch] gh/shunting314/245/orig -> origin/gh/shunting314/245/orig 2025-11-03T16:15:22.1609914Z * [new branch] gh/shunting314/246/base -> origin/gh/shunting314/246/base 2025-11-03T16:15:22.1610114Z * [new branch] gh/shunting314/246/head -> origin/gh/shunting314/246/head 2025-11-03T16:15:22.1611577Z * [new branch] gh/shunting314/246/orig -> origin/gh/shunting314/246/orig 2025-11-03T16:15:22.1611878Z * [new branch] gh/shunting314/247/base -> origin/gh/shunting314/247/base 2025-11-03T16:15:22.1612264Z * [new branch] gh/shunting314/247/head -> origin/gh/shunting314/247/head 2025-11-03T16:15:22.1613978Z * [new branch] gh/shunting314/247/orig -> origin/gh/shunting314/247/orig 2025-11-03T16:15:22.1614290Z * [new branch] gh/shunting314/248/base -> origin/gh/shunting314/248/base 2025-11-03T16:15:22.1616762Z * [new branch] gh/shunting314/248/head -> origin/gh/shunting314/248/head 2025-11-03T16:15:22.1616936Z * [new branch] gh/shunting314/248/orig -> origin/gh/shunting314/248/orig 2025-11-03T16:15:22.1617081Z * [new branch] gh/shunting314/249/base -> origin/gh/shunting314/249/base 2025-11-03T16:15:22.1617243Z * [new branch] gh/shunting314/249/head -> origin/gh/shunting314/249/head 2025-11-03T16:15:22.1617735Z * [new branch] gh/shunting314/249/orig -> origin/gh/shunting314/249/orig 2025-11-03T16:15:22.1619803Z * [new branch] gh/shunting314/250/base -> origin/gh/shunting314/250/base 2025-11-03T16:15:22.1620155Z * [new branch] gh/shunting314/250/head -> origin/gh/shunting314/250/head 2025-11-03T16:15:22.1620377Z * [new branch] gh/shunting314/250/orig -> origin/gh/shunting314/250/orig 2025-11-03T16:15:22.1621861Z * [new branch] gh/shunting314/251/base -> origin/gh/shunting314/251/base 2025-11-03T16:15:22.1622182Z * [new branch] gh/shunting314/251/head -> origin/gh/shunting314/251/head 2025-11-03T16:15:22.1622405Z * [new branch] gh/shunting314/251/orig -> origin/gh/shunting314/251/orig 2025-11-03T16:15:22.1624618Z * [new branch] gh/shunting314/252/base -> origin/gh/shunting314/252/base 2025-11-03T16:15:22.1625152Z * [new branch] gh/shunting314/252/head -> origin/gh/shunting314/252/head 2025-11-03T16:15:22.1625437Z * [new branch] gh/shunting314/252/orig -> origin/gh/shunting314/252/orig 2025-11-03T16:15:22.1625671Z * [new branch] gh/shunting314/253/base -> origin/gh/shunting314/253/base 2025-11-03T16:15:22.1626081Z * [new branch] gh/shunting314/253/head -> origin/gh/shunting314/253/head 2025-11-03T16:15:22.1627066Z * [new branch] gh/shunting314/253/orig -> origin/gh/shunting314/253/orig 2025-11-03T16:15:22.1628189Z * [new branch] gh/shunting314/254/base -> origin/gh/shunting314/254/base 2025-11-03T16:15:22.1628759Z * [new branch] gh/shunting314/254/head -> origin/gh/shunting314/254/head 2025-11-03T16:15:22.1629433Z * [new branch] gh/shunting314/254/orig -> origin/gh/shunting314/254/orig 2025-11-03T16:15:22.1630408Z * [new branch] gh/shunting314/255/base -> origin/gh/shunting314/255/base 2025-11-03T16:15:22.1630694Z * [new branch] gh/shunting314/255/head -> origin/gh/shunting314/255/head 2025-11-03T16:15:22.1632744Z * [new branch] gh/shunting314/255/orig -> origin/gh/shunting314/255/orig 2025-11-03T16:15:22.1632947Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-11-03T16:15:22.1633085Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-11-03T16:15:22.1634477Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-11-03T16:15:22.1637514Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-11-03T16:15:22.1637674Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-11-03T16:15:22.1637807Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-11-03T16:15:22.1637955Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-11-03T16:15:22.1638200Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-11-03T16:15:22.1642261Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-11-03T16:15:22.1642805Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-11-03T16:15:22.1643032Z * [new branch] gh/slayton58/14/base -> origin/gh/slayton58/14/base 2025-11-03T16:15:22.1643652Z * [new branch] gh/slayton58/14/head -> origin/gh/slayton58/14/head 2025-11-03T16:15:22.1643818Z * [new branch] gh/slayton58/14/orig -> origin/gh/slayton58/14/orig 2025-11-03T16:15:22.1643947Z * [new branch] gh/slayton58/16/base -> origin/gh/slayton58/16/base 2025-11-03T16:15:22.1644070Z * [new branch] gh/slayton58/16/head -> origin/gh/slayton58/16/head 2025-11-03T16:15:22.1644213Z * [new branch] gh/slayton58/16/orig -> origin/gh/slayton58/16/orig 2025-11-03T16:15:22.1644424Z * [new branch] gh/slayton58/17/base -> origin/gh/slayton58/17/base 2025-11-03T16:15:22.1645677Z * [new branch] gh/slayton58/17/head -> origin/gh/slayton58/17/head 2025-11-03T16:15:22.1645960Z * [new branch] gh/slayton58/17/orig -> origin/gh/slayton58/17/orig 2025-11-03T16:15:22.1648210Z * [new branch] gh/slayton58/19/base -> origin/gh/slayton58/19/base 2025-11-03T16:15:22.1648523Z * [new branch] gh/slayton58/19/head -> origin/gh/slayton58/19/head 2025-11-03T16:15:22.1648746Z * [new branch] gh/slayton58/19/orig -> origin/gh/slayton58/19/orig 2025-11-03T16:15:22.1648884Z * [new branch] gh/slayton58/20/base -> origin/gh/slayton58/20/base 2025-11-03T16:15:22.1649606Z * [new branch] gh/slayton58/20/head -> origin/gh/slayton58/20/head 2025-11-03T16:15:22.1650380Z * [new branch] gh/slayton58/20/orig -> origin/gh/slayton58/20/orig 2025-11-03T16:15:22.1650773Z * [new branch] gh/slayton58/21/base -> origin/gh/slayton58/21/base 2025-11-03T16:15:22.1652208Z * [new branch] gh/slayton58/21/head -> origin/gh/slayton58/21/head 2025-11-03T16:15:22.1652518Z * [new branch] gh/slayton58/21/orig -> origin/gh/slayton58/21/orig 2025-11-03T16:15:22.1652776Z * [new branch] gh/slayton58/23/base -> origin/gh/slayton58/23/base 2025-11-03T16:15:22.1654074Z * [new branch] gh/slayton58/23/head -> origin/gh/slayton58/23/head 2025-11-03T16:15:22.1654339Z * [new branch] gh/slayton58/23/orig -> origin/gh/slayton58/23/orig 2025-11-03T16:15:22.1656791Z * [new branch] gh/slayton58/25/base -> origin/gh/slayton58/25/base 2025-11-03T16:15:22.1657108Z * [new branch] gh/slayton58/25/head -> origin/gh/slayton58/25/head 2025-11-03T16:15:22.1657339Z * [new branch] gh/slayton58/25/orig -> origin/gh/slayton58/25/orig 2025-11-03T16:15:22.1657548Z * [new branch] gh/slayton58/26/base -> origin/gh/slayton58/26/base 2025-11-03T16:15:22.1657704Z * [new branch] gh/slayton58/26/head -> origin/gh/slayton58/26/head 2025-11-03T16:15:22.1659057Z * [new branch] gh/slayton58/26/orig -> origin/gh/slayton58/26/orig 2025-11-03T16:15:22.1659253Z * [new branch] gh/slayton58/27/base -> origin/gh/slayton58/27/base 2025-11-03T16:15:22.1660756Z * [new branch] gh/slayton58/27/head -> origin/gh/slayton58/27/head 2025-11-03T16:15:22.1661049Z * [new branch] gh/slayton58/27/orig -> origin/gh/slayton58/27/orig 2025-11-03T16:15:22.1661353Z * [new branch] gh/slayton58/28/base -> origin/gh/slayton58/28/base 2025-11-03T16:15:22.1662965Z * [new branch] gh/slayton58/28/head -> origin/gh/slayton58/28/head 2025-11-03T16:15:22.1663286Z * [new branch] gh/slayton58/28/orig -> origin/gh/slayton58/28/orig 2025-11-03T16:15:22.1663605Z * [new branch] gh/slayton58/29/base -> origin/gh/slayton58/29/base 2025-11-03T16:15:22.1664228Z * [new branch] gh/slayton58/29/head -> origin/gh/slayton58/29/head 2025-11-03T16:15:22.1667638Z * [new branch] gh/slayton58/29/head-jeffdaily-patch-1 -> origin/gh/slayton58/29/head-jeffdaily-patch-1 2025-11-03T16:15:22.1668008Z * [new branch] gh/slayton58/29/head-jeffdaily-patch-2 -> origin/gh/slayton58/29/head-jeffdaily-patch-2 2025-11-03T16:15:22.1668291Z * [new branch] gh/slayton58/29/orig -> origin/gh/slayton58/29/orig 2025-11-03T16:15:22.1668523Z * [new branch] gh/slayton58/30/base -> origin/gh/slayton58/30/base 2025-11-03T16:15:22.1668679Z * [new branch] gh/slayton58/30/head -> origin/gh/slayton58/30/head 2025-11-03T16:15:22.1669181Z * [new branch] gh/slayton58/30/orig -> origin/gh/slayton58/30/orig 2025-11-03T16:15:22.1670634Z * [new branch] gh/slayton58/31/base -> origin/gh/slayton58/31/base 2025-11-03T16:15:22.1670798Z * [new branch] gh/slayton58/31/head -> origin/gh/slayton58/31/head 2025-11-03T16:15:22.1671175Z * [new branch] gh/slayton58/31/orig -> origin/gh/slayton58/31/orig 2025-11-03T16:15:22.1672853Z * [new branch] gh/slayton58/32/base -> origin/gh/slayton58/32/base 2025-11-03T16:15:22.1673030Z * [new branch] gh/slayton58/32/head -> origin/gh/slayton58/32/head 2025-11-03T16:15:22.1673529Z * [new branch] gh/slayton58/32/orig -> origin/gh/slayton58/32/orig 2025-11-03T16:15:22.1674614Z * [new branch] gh/slayton58/33/base -> origin/gh/slayton58/33/base 2025-11-03T16:15:22.1675108Z * [new branch] gh/slayton58/33/head -> origin/gh/slayton58/33/head 2025-11-03T16:15:22.1677368Z * [new branch] gh/slayton58/33/orig -> origin/gh/slayton58/33/orig 2025-11-03T16:15:22.1677666Z * [new branch] gh/slayton58/34/base -> origin/gh/slayton58/34/base 2025-11-03T16:15:22.1677905Z * [new branch] gh/slayton58/34/head -> origin/gh/slayton58/34/head 2025-11-03T16:15:22.1678333Z * [new branch] gh/slayton58/34/orig -> origin/gh/slayton58/34/orig 2025-11-03T16:15:22.1681884Z * [new branch] gh/slayton58/35/base -> origin/gh/slayton58/35/base 2025-11-03T16:15:22.1682188Z * [new branch] gh/slayton58/35/head -> origin/gh/slayton58/35/head 2025-11-03T16:15:22.1682410Z * [new branch] gh/slayton58/35/orig -> origin/gh/slayton58/35/orig 2025-11-03T16:15:22.1682631Z * [new branch] gh/slayton58/36/base -> origin/gh/slayton58/36/base 2025-11-03T16:15:22.1684005Z * [new branch] gh/slayton58/36/head -> origin/gh/slayton58/36/head 2025-11-03T16:15:22.1684273Z * [new branch] gh/slayton58/36/orig -> origin/gh/slayton58/36/orig 2025-11-03T16:15:22.1687706Z * [new branch] gh/slayton58/37/base -> origin/gh/slayton58/37/base 2025-11-03T16:15:22.1688042Z * [new branch] gh/slayton58/37/head -> origin/gh/slayton58/37/head 2025-11-03T16:15:22.1688259Z * [new branch] gh/slayton58/37/orig -> origin/gh/slayton58/37/orig 2025-11-03T16:15:22.1688408Z * [new branch] gh/slayton58/38/base -> origin/gh/slayton58/38/base 2025-11-03T16:15:22.1688615Z * [new branch] gh/slayton58/38/head -> origin/gh/slayton58/38/head 2025-11-03T16:15:22.1689211Z * [new branch] gh/slayton58/38/orig -> origin/gh/slayton58/38/orig 2025-11-03T16:15:22.1689403Z * [new branch] gh/slayton58/39/base -> origin/gh/slayton58/39/base 2025-11-03T16:15:22.1690926Z * [new branch] gh/slayton58/39/head -> origin/gh/slayton58/39/head 2025-11-03T16:15:22.1691256Z * [new branch] gh/slayton58/39/orig -> origin/gh/slayton58/39/orig 2025-11-03T16:15:22.1691691Z * [new branch] gh/slayton58/40/base -> origin/gh/slayton58/40/base 2025-11-03T16:15:22.1693670Z * [new branch] gh/slayton58/40/head -> origin/gh/slayton58/40/head 2025-11-03T16:15:22.1693834Z * [new branch] gh/slayton58/40/orig -> origin/gh/slayton58/40/orig 2025-11-03T16:15:22.1693965Z * [new branch] gh/slayton58/41/base -> origin/gh/slayton58/41/base 2025-11-03T16:15:22.1696039Z * [new branch] gh/slayton58/41/head -> origin/gh/slayton58/41/head 2025-11-03T16:15:22.1696354Z * [new branch] gh/slayton58/41/orig -> origin/gh/slayton58/41/orig 2025-11-03T16:15:22.1696582Z * [new branch] gh/slayton58/42/base -> origin/gh/slayton58/42/base 2025-11-03T16:15:22.1696739Z * [new branch] gh/slayton58/42/head -> origin/gh/slayton58/42/head 2025-11-03T16:15:22.1698347Z * [new branch] gh/slayton58/42/orig -> origin/gh/slayton58/42/orig 2025-11-03T16:15:22.1698557Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-11-03T16:15:22.1700749Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-11-03T16:15:22.1701062Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-11-03T16:15:22.1701301Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-11-03T16:15:22.1705212Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-11-03T16:15:22.1705534Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-11-03T16:15:22.1705895Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-11-03T16:15:22.1706161Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-11-03T16:15:22.1706381Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-11-03T16:15:22.1706945Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-11-03T16:15:22.1707105Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-11-03T16:15:22.1707398Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-11-03T16:15:22.1708704Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-11-03T16:15:22.1709199Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-11-03T16:15:22.1711792Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-11-03T16:15:22.1711988Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-11-03T16:15:22.1712131Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-11-03T16:15:22.1712287Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-11-03T16:15:22.1712972Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-11-03T16:15:22.1714130Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-11-03T16:15:22.1714628Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-11-03T16:15:22.1717980Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-11-03T16:15:22.1718284Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-11-03T16:15:22.1718506Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-11-03T16:15:22.1718662Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-11-03T16:15:22.1718891Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-11-03T16:15:22.1719227Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-11-03T16:15:22.1719991Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-11-03T16:15:22.1723364Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-11-03T16:15:22.1723673Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-11-03T16:15:22.1723886Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-11-03T16:15:22.1724098Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-11-03T16:15:22.1724256Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-11-03T16:15:22.1724494Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-11-03T16:15:22.1725045Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-11-03T16:15:22.1725674Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-11-03T16:15:22.1728839Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-11-03T16:15:22.1729167Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-11-03T16:15:22.1729382Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-11-03T16:15:22.1729533Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-11-03T16:15:22.1729755Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-11-03T16:15:22.1731154Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-11-03T16:15:22.1731498Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-11-03T16:15:22.1733341Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-11-03T16:15:22.1733655Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-11-03T16:15:22.1733878Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-11-03T16:15:22.1734029Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-11-03T16:15:22.1735639Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-11-03T16:15:22.1735951Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-11-03T16:15:22.1736168Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-11-03T16:15:22.1738170Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-11-03T16:15:22.1738489Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-11-03T16:15:22.1738920Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-11-03T16:15:22.1740213Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-11-03T16:15:22.1740481Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-11-03T16:15:22.1742503Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-11-03T16:15:22.1742817Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-11-03T16:15:22.1743031Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-11-03T16:15:22.1743460Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-11-03T16:15:22.1744272Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-11-03T16:15:22.1744736Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-11-03T16:15:22.1746489Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-11-03T16:15:22.1746804Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-11-03T16:15:22.1747021Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-11-03T16:15:22.1748264Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-11-03T16:15:22.1748588Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-11-03T16:15:22.1748846Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-11-03T16:15:22.1750875Z * [new branch] gh/soulitzer/381/base -> origin/gh/soulitzer/381/base 2025-11-03T16:15:22.1751049Z * [new branch] gh/soulitzer/381/head -> origin/gh/soulitzer/381/head 2025-11-03T16:15:22.1751181Z * [new branch] gh/soulitzer/381/orig -> origin/gh/soulitzer/381/orig 2025-11-03T16:15:22.1752095Z * [new branch] gh/soulitzer/384/base -> origin/gh/soulitzer/384/base 2025-11-03T16:15:22.1752503Z * [new branch] gh/soulitzer/384/head -> origin/gh/soulitzer/384/head 2025-11-03T16:15:22.1753314Z * [new branch] gh/soulitzer/384/orig -> origin/gh/soulitzer/384/orig 2025-11-03T16:15:22.1757769Z * [new branch] gh/soulitzer/385/base -> origin/gh/soulitzer/385/base 2025-11-03T16:15:22.1758098Z * [new branch] gh/soulitzer/385/head -> origin/gh/soulitzer/385/head 2025-11-03T16:15:22.1758331Z * [new branch] gh/soulitzer/385/orig -> origin/gh/soulitzer/385/orig 2025-11-03T16:15:22.1758625Z * [new branch] gh/soulitzer/386/base -> origin/gh/soulitzer/386/base 2025-11-03T16:15:22.1758774Z * [new branch] gh/soulitzer/386/head -> origin/gh/soulitzer/386/head 2025-11-03T16:15:22.1758915Z * [new branch] gh/soulitzer/386/orig -> origin/gh/soulitzer/386/orig 2025-11-03T16:15:22.1759277Z * [new branch] gh/soulitzer/387/base -> origin/gh/soulitzer/387/base 2025-11-03T16:15:22.1760607Z * [new branch] gh/soulitzer/387/head -> origin/gh/soulitzer/387/head 2025-11-03T16:15:22.1760922Z * [new branch] gh/soulitzer/387/orig -> origin/gh/soulitzer/387/orig 2025-11-03T16:15:22.1763680Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-11-03T16:15:22.1764017Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-11-03T16:15:22.1764245Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-11-03T16:15:22.1764389Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-11-03T16:15:22.1765050Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-11-03T16:15:22.1765245Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-11-03T16:15:22.1766800Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-11-03T16:15:22.1767089Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-11-03T16:15:22.1767471Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-11-03T16:15:22.1769088Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-11-03T16:15:22.1769400Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-11-03T16:15:22.1769630Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-11-03T16:15:22.1771699Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-11-03T16:15:22.1772012Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-11-03T16:15:22.1772442Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-11-03T16:15:22.1772600Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-11-03T16:15:22.1773279Z * [new branch] gh/swolchok/838/base -> origin/gh/swolchok/838/base 2025-11-03T16:15:22.1774162Z * [new branch] gh/swolchok/838/head -> origin/gh/swolchok/838/head 2025-11-03T16:15:22.1774559Z * [new branch] gh/swolchok/838/orig -> origin/gh/swolchok/838/orig 2025-11-03T16:15:22.1776989Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-11-03T16:15:22.1777313Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-11-03T16:15:22.1777541Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-11-03T16:15:22.1777692Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-11-03T16:15:22.1778832Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-11-03T16:15:22.1779321Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-11-03T16:15:22.1781217Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-11-03T16:15:22.1781547Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-11-03T16:15:22.1781783Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-11-03T16:15:22.1783089Z * [new branch] gh/swolchok/843/base -> origin/gh/swolchok/843/base 2025-11-03T16:15:22.1783574Z * [new branch] gh/swolchok/843/head -> origin/gh/swolchok/843/head 2025-11-03T16:15:22.1783837Z * [new branch] gh/swolchok/843/orig -> origin/gh/swolchok/843/orig 2025-11-03T16:15:22.1785996Z * [new branch] gh/swolchok/844/base -> origin/gh/swolchok/844/base 2025-11-03T16:15:22.1786160Z * [new branch] gh/swolchok/844/head -> origin/gh/swolchok/844/head 2025-11-03T16:15:22.1786307Z * [new branch] gh/swolchok/844/orig -> origin/gh/swolchok/844/orig 2025-11-03T16:15:22.1788240Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-11-03T16:15:22.1788397Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-11-03T16:15:22.1788528Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-11-03T16:15:22.1789581Z * [new branch] gh/swolchok/847/base -> origin/gh/swolchok/847/base 2025-11-03T16:15:22.1790385Z * [new branch] gh/swolchok/847/head -> origin/gh/swolchok/847/head 2025-11-03T16:15:22.1790706Z * [new branch] gh/swolchok/847/orig -> origin/gh/swolchok/847/orig 2025-11-03T16:15:22.1791787Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-11-03T16:15:22.1792159Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-11-03T16:15:22.1794166Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-11-03T16:15:22.1794339Z * [new branch] gh/swolchok/849/base -> origin/gh/swolchok/849/base 2025-11-03T16:15:22.1794524Z * [new branch] gh/swolchok/849/head -> origin/gh/swolchok/849/head 2025-11-03T16:15:22.1798203Z * [new branch] gh/swolchok/849/orig -> origin/gh/swolchok/849/orig 2025-11-03T16:15:22.1798534Z * [new branch] gh/swolchok/850/base -> origin/gh/swolchok/850/base 2025-11-03T16:15:22.1798758Z * [new branch] gh/swolchok/850/head -> origin/gh/swolchok/850/head 2025-11-03T16:15:22.1798908Z * [new branch] gh/swolchok/850/orig -> origin/gh/swolchok/850/orig 2025-11-03T16:15:22.1799298Z * [new branch] gh/swolchok/851/base -> origin/gh/swolchok/851/base 2025-11-03T16:15:22.1799427Z * [new branch] gh/swolchok/851/head -> origin/gh/swolchok/851/head 2025-11-03T16:15:22.1799888Z * [new branch] gh/swolchok/851/orig -> origin/gh/swolchok/851/orig 2025-11-03T16:15:22.1801582Z * [new branch] gh/swolchok/852/base -> origin/gh/swolchok/852/base 2025-11-03T16:15:22.1801903Z * [new branch] gh/swolchok/852/head -> origin/gh/swolchok/852/head 2025-11-03T16:15:22.1802158Z * [new branch] gh/swolchok/852/orig -> origin/gh/swolchok/852/orig 2025-11-03T16:15:22.1803919Z * [new branch] gh/swolchok/853/base -> origin/gh/swolchok/853/base 2025-11-03T16:15:22.1804234Z * [new branch] gh/swolchok/853/head -> origin/gh/swolchok/853/head 2025-11-03T16:15:22.1804459Z * [new branch] gh/swolchok/853/orig -> origin/gh/swolchok/853/orig 2025-11-03T16:15:22.1806175Z * [new branch] gh/swolchok/854/base -> origin/gh/swolchok/854/base 2025-11-03T16:15:22.1806488Z * [new branch] gh/swolchok/854/head -> origin/gh/swolchok/854/head 2025-11-03T16:15:22.1806701Z * [new branch] gh/swolchok/854/orig -> origin/gh/swolchok/854/orig 2025-11-03T16:15:22.1808937Z * [new branch] gh/swolchok/855/base -> origin/gh/swolchok/855/base 2025-11-03T16:15:22.1809245Z * [new branch] gh/swolchok/855/head -> origin/gh/swolchok/855/head 2025-11-03T16:15:22.1809464Z * [new branch] gh/swolchok/855/orig -> origin/gh/swolchok/855/orig 2025-11-03T16:15:22.1809741Z * [new branch] gh/swolchok/856/base -> origin/gh/swolchok/856/base 2025-11-03T16:15:22.1810371Z * [new branch] gh/swolchok/856/head -> origin/gh/swolchok/856/head 2025-11-03T16:15:22.1812930Z * [new branch] gh/swolchok/856/orig -> origin/gh/swolchok/856/orig 2025-11-03T16:15:22.1813426Z * [new branch] gh/swolchok/857/base -> origin/gh/swolchok/857/base 2025-11-03T16:15:22.1813687Z * [new branch] gh/swolchok/857/head -> origin/gh/swolchok/857/head 2025-11-03T16:15:22.1813835Z * [new branch] gh/swolchok/857/orig -> origin/gh/swolchok/857/orig 2025-11-03T16:15:22.1815196Z * [new branch] gh/swolchok/858/base -> origin/gh/swolchok/858/base 2025-11-03T16:15:22.1815454Z * [new branch] gh/swolchok/858/head -> origin/gh/swolchok/858/head 2025-11-03T16:15:22.1815877Z * [new branch] gh/swolchok/858/orig -> origin/gh/swolchok/858/orig 2025-11-03T16:15:22.1817536Z * [new branch] gh/swolchok/859/base -> origin/gh/swolchok/859/base 2025-11-03T16:15:22.1817839Z * [new branch] gh/swolchok/859/head -> origin/gh/swolchok/859/head 2025-11-03T16:15:22.1818053Z * [new branch] gh/swolchok/859/orig -> origin/gh/swolchok/859/orig 2025-11-03T16:15:22.1820996Z * [new branch] gh/swolchok/860/base -> origin/gh/swolchok/860/base 2025-11-03T16:15:22.1821167Z * [new branch] gh/swolchok/860/head -> origin/gh/swolchok/860/head 2025-11-03T16:15:22.1821299Z * [new branch] gh/swolchok/860/orig -> origin/gh/swolchok/860/orig 2025-11-03T16:15:22.1821466Z * [new branch] gh/swolchok/861/base -> origin/gh/swolchok/861/base 2025-11-03T16:15:22.1822933Z * [new branch] gh/swolchok/861/head -> origin/gh/swolchok/861/head 2025-11-03T16:15:22.1823113Z * [new branch] gh/swolchok/861/orig -> origin/gh/swolchok/861/orig 2025-11-03T16:15:22.1825623Z * [new branch] gh/swolchok/862/base -> origin/gh/swolchok/862/base 2025-11-03T16:15:22.1825955Z * [new branch] gh/swolchok/862/head -> origin/gh/swolchok/862/head 2025-11-03T16:15:22.1826455Z * [new branch] gh/swolchok/862/orig -> origin/gh/swolchok/862/orig 2025-11-03T16:15:22.1826600Z * [new branch] gh/swolchok/863/base -> origin/gh/swolchok/863/base 2025-11-03T16:15:22.1826840Z * [new branch] gh/swolchok/863/head -> origin/gh/swolchok/863/head 2025-11-03T16:15:22.1828178Z * [new branch] gh/swolchok/863/orig -> origin/gh/swolchok/863/orig 2025-11-03T16:15:22.1829136Z * [new branch] gh/swolchok/864/base -> origin/gh/swolchok/864/base 2025-11-03T16:15:22.1829774Z * [new branch] gh/swolchok/864/head -> origin/gh/swolchok/864/head 2025-11-03T16:15:22.1830016Z * [new branch] gh/swolchok/864/orig -> origin/gh/swolchok/864/orig 2025-11-03T16:15:22.1831408Z * [new branch] gh/swolchok/865/base -> origin/gh/swolchok/865/base 2025-11-03T16:15:22.1831593Z * [new branch] gh/swolchok/865/head -> origin/gh/swolchok/865/head 2025-11-03T16:15:22.1832179Z * [new branch] gh/swolchok/865/orig -> origin/gh/swolchok/865/orig 2025-11-03T16:15:22.1835909Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-11-03T16:15:22.1836167Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-11-03T16:15:22.1841140Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-11-03T16:15:22.1841469Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-11-03T16:15:22.1841616Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-11-03T16:15:22.1842014Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-11-03T16:15:22.1842307Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-11-03T16:15:22.1842893Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-11-03T16:15:22.1843082Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-11-03T16:15:22.1843224Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-11-03T16:15:22.1843363Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-11-03T16:15:22.1843514Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-11-03T16:15:22.1843654Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-11-03T16:15:22.1843812Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-11-03T16:15:22.1848098Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-11-03T16:15:22.1848436Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-11-03T16:15:22.1848704Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-11-03T16:15:22.1848956Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-11-03T16:15:22.1849178Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-11-03T16:15:22.1850137Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-11-03T16:15:22.1850350Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-11-03T16:15:22.1850569Z * [new branch] gh/tugsbayasgalan/3/base -> origin/gh/tugsbayasgalan/3/base 2025-11-03T16:15:22.1850745Z * [new branch] gh/tugsbayasgalan/3/head -> origin/gh/tugsbayasgalan/3/head 2025-11-03T16:15:22.1850885Z * [new branch] gh/tugsbayasgalan/3/orig -> origin/gh/tugsbayasgalan/3/orig 2025-11-03T16:15:22.1851206Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-11-03T16:15:22.1851462Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-11-03T16:15:22.1856796Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-11-03T16:15:22.1857143Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-11-03T16:15:22.1857400Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-11-03T16:15:22.1857629Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-11-03T16:15:22.1857883Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-11-03T16:15:22.1858157Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-11-03T16:15:22.1858322Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-11-03T16:15:22.1858919Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-11-03T16:15:22.1859069Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-11-03T16:15:22.1859217Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-11-03T16:15:22.1859367Z * [new branch] gh/tugsbayasgalan/39/base -> origin/gh/tugsbayasgalan/39/base 2025-11-03T16:15:22.1859515Z * [new branch] gh/tugsbayasgalan/39/head -> origin/gh/tugsbayasgalan/39/head 2025-11-03T16:15:22.1859664Z * [new branch] gh/tugsbayasgalan/39/orig -> origin/gh/tugsbayasgalan/39/orig 2025-11-03T16:15:22.1864467Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-11-03T16:15:22.1864817Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-11-03T16:15:22.1865083Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-11-03T16:15:22.1865298Z * [new branch] gh/tugsbayasgalan/46/base -> origin/gh/tugsbayasgalan/46/base 2025-11-03T16:15:22.1865506Z * [new branch] gh/tugsbayasgalan/46/head -> origin/gh/tugsbayasgalan/46/head 2025-11-03T16:15:22.1866142Z * [new branch] gh/tugsbayasgalan/46/orig -> origin/gh/tugsbayasgalan/46/orig 2025-11-03T16:15:22.1866326Z * [new branch] gh/tugsbayasgalan/47/base -> origin/gh/tugsbayasgalan/47/base 2025-11-03T16:15:22.1866471Z * [new branch] gh/tugsbayasgalan/47/head -> origin/gh/tugsbayasgalan/47/head 2025-11-03T16:15:22.1866643Z * [new branch] gh/tugsbayasgalan/47/orig -> origin/gh/tugsbayasgalan/47/orig 2025-11-03T16:15:22.1866783Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-11-03T16:15:22.1866946Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-11-03T16:15:22.1867266Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-11-03T16:15:22.1867757Z * [new branch] gh/tugsbayasgalan/49/base -> origin/gh/tugsbayasgalan/49/base 2025-11-03T16:15:22.1868781Z * [new branch] gh/tugsbayasgalan/49/head -> origin/gh/tugsbayasgalan/49/head 2025-11-03T16:15:22.1869372Z * [new branch] gh/tugsbayasgalan/49/orig -> origin/gh/tugsbayasgalan/49/orig 2025-11-03T16:15:22.1870760Z * [new branch] gh/tugsbayasgalan/50/base -> origin/gh/tugsbayasgalan/50/base 2025-11-03T16:15:22.1871072Z * [new branch] gh/tugsbayasgalan/50/head -> origin/gh/tugsbayasgalan/50/head 2025-11-03T16:15:22.1872038Z * [new branch] gh/tugsbayasgalan/50/orig -> origin/gh/tugsbayasgalan/50/orig 2025-11-03T16:15:22.1873094Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-11-03T16:15:22.1873389Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-11-03T16:15:22.1877915Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-11-03T16:15:22.1878246Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-11-03T16:15:22.1878504Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-11-03T16:15:22.1878668Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-11-03T16:15:22.1878820Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-11-03T16:15:22.1879100Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-11-03T16:15:22.1879690Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-11-03T16:15:22.1880019Z * [new branch] gh/tugsbayasgalan/54/base -> origin/gh/tugsbayasgalan/54/base 2025-11-03T16:15:22.1880561Z * [new branch] gh/tugsbayasgalan/54/head -> origin/gh/tugsbayasgalan/54/head 2025-11-03T16:15:22.1882059Z * [new branch] gh/tugsbayasgalan/54/orig -> origin/gh/tugsbayasgalan/54/orig 2025-11-03T16:15:22.1882391Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2025-11-03T16:15:22.1883685Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2025-11-03T16:15:22.1884028Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2025-11-03T16:15:22.1890846Z * [new branch] gh/tugsbayasgalan/56/base -> origin/gh/tugsbayasgalan/56/base 2025-11-03T16:15:22.1891191Z * [new branch] gh/tugsbayasgalan/56/head -> origin/gh/tugsbayasgalan/56/head 2025-11-03T16:15:22.1891467Z * [new branch] gh/tugsbayasgalan/56/orig -> origin/gh/tugsbayasgalan/56/orig 2025-11-03T16:15:22.1891693Z * [new branch] gh/tugsbayasgalan/57/base -> origin/gh/tugsbayasgalan/57/base 2025-11-03T16:15:22.1891860Z * [new branch] gh/tugsbayasgalan/57/head -> origin/gh/tugsbayasgalan/57/head 2025-11-03T16:15:22.1892457Z * [new branch] gh/tugsbayasgalan/57/orig -> origin/gh/tugsbayasgalan/57/orig 2025-11-03T16:15:22.1892655Z * [new branch] gh/tugsbayasgalan/58/base -> origin/gh/tugsbayasgalan/58/base 2025-11-03T16:15:22.1893175Z * [new branch] gh/tugsbayasgalan/58/head -> origin/gh/tugsbayasgalan/58/head 2025-11-03T16:15:22.1894153Z * [new branch] gh/tugsbayasgalan/58/orig -> origin/gh/tugsbayasgalan/58/orig 2025-11-03T16:15:22.1896767Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2025-11-03T16:15:22.1897090Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2025-11-03T16:15:22.1897367Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2025-11-03T16:15:22.1897629Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-11-03T16:15:22.1897871Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-11-03T16:15:22.1898684Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-11-03T16:15:22.1901838Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2025-11-03T16:15:22.1902174Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2025-11-03T16:15:22.1902424Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2025-11-03T16:15:22.1902663Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2025-11-03T16:15:22.1903083Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2025-11-03T16:15:22.1903682Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2025-11-03T16:15:22.1903855Z * [new branch] gh/tugsbayasgalan/62/base -> origin/gh/tugsbayasgalan/62/base 2025-11-03T16:15:22.1904073Z * [new branch] gh/tugsbayasgalan/62/head -> origin/gh/tugsbayasgalan/62/head 2025-11-03T16:15:22.1905218Z * [new branch] gh/tugsbayasgalan/62/orig -> origin/gh/tugsbayasgalan/62/orig 2025-11-03T16:15:22.1905609Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2025-11-03T16:15:22.1906154Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2025-11-03T16:15:22.1909947Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2025-11-03T16:15:22.1910281Z * [new branch] gh/tugsbayasgalan/64/base -> origin/gh/tugsbayasgalan/64/base 2025-11-03T16:15:22.1910559Z * [new branch] gh/tugsbayasgalan/64/head -> origin/gh/tugsbayasgalan/64/head 2025-11-03T16:15:22.1910790Z * [new branch] gh/tugsbayasgalan/64/orig -> origin/gh/tugsbayasgalan/64/orig 2025-11-03T16:15:22.1911030Z * [new branch] gh/tugsbayasgalan/65/base -> origin/gh/tugsbayasgalan/65/base 2025-11-03T16:15:22.1911186Z * [new branch] gh/tugsbayasgalan/65/head -> origin/gh/tugsbayasgalan/65/head 2025-11-03T16:15:22.1912210Z * [new branch] gh/tugsbayasgalan/65/orig -> origin/gh/tugsbayasgalan/65/orig 2025-11-03T16:15:22.1912717Z * [new branch] gh/tugsbayasgalan/66/base -> origin/gh/tugsbayasgalan/66/base 2025-11-03T16:15:22.1913741Z * [new branch] gh/tugsbayasgalan/66/head -> origin/gh/tugsbayasgalan/66/head 2025-11-03T16:15:22.1918466Z * [new branch] gh/tugsbayasgalan/66/orig -> origin/gh/tugsbayasgalan/66/orig 2025-11-03T16:15:22.1918760Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2025-11-03T16:15:22.1924317Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2025-11-03T16:15:22.1926103Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2025-11-03T16:15:22.1926475Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2025-11-03T16:15:22.1926701Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2025-11-03T16:15:22.1926876Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2025-11-03T16:15:22.1927027Z * [new branch] gh/tugsbayasgalan/69/base -> origin/gh/tugsbayasgalan/69/base 2025-11-03T16:15:22.1927164Z * [new branch] gh/tugsbayasgalan/69/head -> origin/gh/tugsbayasgalan/69/head 2025-11-03T16:15:22.1927317Z * [new branch] gh/tugsbayasgalan/69/orig -> origin/gh/tugsbayasgalan/69/orig 2025-11-03T16:15:22.1927468Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-11-03T16:15:22.1931272Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-11-03T16:15:22.1931596Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-11-03T16:15:22.1931857Z * [new branch] gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base 2025-11-03T16:15:22.1932074Z * [new branch] gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head 2025-11-03T16:15:22.1932355Z * [new branch] gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig 2025-11-03T16:15:22.1932957Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2025-11-03T16:15:22.1933134Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2025-11-03T16:15:22.1933469Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2025-11-03T16:15:22.1933627Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-11-03T16:15:22.1933774Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-11-03T16:15:22.1938388Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-11-03T16:15:22.1938709Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-11-03T16:15:22.1938931Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-11-03T16:15:22.1939112Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-11-03T16:15:22.1939341Z * [new branch] gh/v0i0/13/base -> origin/gh/v0i0/13/base 2025-11-03T16:15:22.1939483Z * [new branch] gh/v0i0/13/head -> origin/gh/v0i0/13/head 2025-11-03T16:15:22.1939666Z * [new branch] gh/v0i0/13/orig -> origin/gh/v0i0/13/orig 2025-11-03T16:15:22.1940345Z * [new branch] gh/v0i0/14/base -> origin/gh/v0i0/14/base 2025-11-03T16:15:22.1940499Z * [new branch] gh/v0i0/14/head -> origin/gh/v0i0/14/head 2025-11-03T16:15:22.1940627Z * [new branch] gh/v0i0/14/orig -> origin/gh/v0i0/14/orig 2025-11-03T16:15:22.1945884Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-11-03T16:15:22.1946202Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-11-03T16:15:22.1946557Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-11-03T16:15:22.1946827Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-11-03T16:15:22.1947053Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-11-03T16:15:22.1947205Z * [new branch] gh/vishal9-team/3/base -> origin/gh/vishal9-team/3/base 2025-11-03T16:15:22.1947890Z * [new branch] gh/vishal9-team/3/head -> origin/gh/vishal9-team/3/head 2025-11-03T16:15:22.1948076Z * [new branch] gh/vishal9-team/3/orig -> origin/gh/vishal9-team/3/orig 2025-11-03T16:15:22.1948216Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-11-03T16:15:22.1948354Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-11-03T16:15:22.1948478Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-11-03T16:15:22.1948831Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-11-03T16:15:22.1948987Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-11-03T16:15:22.1949711Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-11-03T16:15:22.1950854Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-11-03T16:15:22.1951097Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-11-03T16:15:22.1952093Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-11-03T16:15:22.1952634Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-11-03T16:15:22.1953539Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-11-03T16:15:22.1954211Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-11-03T16:15:22.1955556Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-11-03T16:15:22.1955847Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-11-03T16:15:22.1956307Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-11-03T16:15:22.1960986Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-11-03T16:15:22.1961156Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-11-03T16:15:22.1961287Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-11-03T16:15:22.1961419Z * [new branch] gh/wconstab/448/base -> origin/gh/wconstab/448/base 2025-11-03T16:15:22.1961557Z * [new branch] gh/wconstab/448/head -> origin/gh/wconstab/448/head 2025-11-03T16:15:22.1961876Z * [new branch] gh/wconstab/448/orig -> origin/gh/wconstab/448/orig 2025-11-03T16:15:22.1966163Z * [new branch] gh/weifengpy/34/base -> origin/gh/weifengpy/34/base 2025-11-03T16:15:22.1966481Z * [new branch] gh/weifengpy/34/head -> origin/gh/weifengpy/34/head 2025-11-03T16:15:22.1966745Z * [new branch] gh/weifengpy/34/orig -> origin/gh/weifengpy/34/orig 2025-11-03T16:15:22.1966892Z * [new branch] gh/weifengpy/35/base -> origin/gh/weifengpy/35/base 2025-11-03T16:15:22.1967133Z * [new branch] gh/weifengpy/35/head -> origin/gh/weifengpy/35/head 2025-11-03T16:15:22.1967282Z * [new branch] gh/weifengpy/35/orig -> origin/gh/weifengpy/35/orig 2025-11-03T16:15:22.1967422Z * [new branch] gh/weifengpy/36/base -> origin/gh/weifengpy/36/base 2025-11-03T16:15:22.1967903Z * [new branch] gh/weifengpy/36/head -> origin/gh/weifengpy/36/head 2025-11-03T16:15:22.1969351Z * [new branch] gh/weifengpy/36/orig -> origin/gh/weifengpy/36/orig 2025-11-03T16:15:22.1969624Z * [new branch] gh/weifengpy/37/base -> origin/gh/weifengpy/37/base 2025-11-03T16:15:22.1972162Z * [new branch] gh/weifengpy/37/head -> origin/gh/weifengpy/37/head 2025-11-03T16:15:22.1972496Z * [new branch] gh/weifengpy/37/orig -> origin/gh/weifengpy/37/orig 2025-11-03T16:15:22.1972717Z * [new branch] gh/weifengpy/38/base -> origin/gh/weifengpy/38/base 2025-11-03T16:15:22.1972875Z * [new branch] gh/weifengpy/38/head -> origin/gh/weifengpy/38/head 2025-11-03T16:15:22.1973377Z * [new branch] gh/weifengpy/38/orig -> origin/gh/weifengpy/38/orig 2025-11-03T16:15:22.1975151Z * [new branch] gh/weifengpy/39/base -> origin/gh/weifengpy/39/base 2025-11-03T16:15:22.1975482Z * [new branch] gh/weifengpy/39/head -> origin/gh/weifengpy/39/head 2025-11-03T16:15:22.1975722Z * [new branch] gh/weifengpy/39/orig -> origin/gh/weifengpy/39/orig 2025-11-03T16:15:22.1977153Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-11-03T16:15:22.1977496Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-11-03T16:15:22.1977795Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-11-03T16:15:22.1982037Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-11-03T16:15:22.1982365Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-11-03T16:15:22.1982597Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-11-03T16:15:22.1982763Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-11-03T16:15:22.1982921Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-11-03T16:15:22.1983206Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-11-03T16:15:22.1983514Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-11-03T16:15:22.1984239Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-11-03T16:15:22.1984983Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-11-03T16:15:22.1988187Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-11-03T16:15:22.1988535Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-11-03T16:15:22.1988791Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-11-03T16:15:22.1989022Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-11-03T16:15:22.1989575Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-11-03T16:15:22.1989750Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-11-03T16:15:22.1990273Z * [new branch] gh/williamwen42/290/base -> origin/gh/williamwen42/290/base 2025-11-03T16:15:22.1990957Z * [new branch] gh/williamwen42/290/head -> origin/gh/williamwen42/290/head 2025-11-03T16:15:22.1991406Z * [new branch] gh/williamwen42/290/orig -> origin/gh/williamwen42/290/orig 2025-11-03T16:15:22.1995249Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-11-03T16:15:22.1995570Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-11-03T16:15:22.1995821Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-11-03T16:15:22.1996108Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-11-03T16:15:22.1996382Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-11-03T16:15:22.1996628Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-11-03T16:15:22.1996851Z * [new branch] gh/williamwen42/305/base -> origin/gh/williamwen42/305/base 2025-11-03T16:15:22.1997233Z * [new branch] gh/williamwen42/305/head -> origin/gh/williamwen42/305/head 2025-11-03T16:15:22.1998449Z * [new branch] gh/williamwen42/305/orig -> origin/gh/williamwen42/305/orig 2025-11-03T16:15:22.1998697Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-11-03T16:15:22.2001061Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-11-03T16:15:22.2001382Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-11-03T16:15:22.2001625Z * [new branch] gh/williamwen42/307/base -> origin/gh/williamwen42/307/base 2025-11-03T16:15:22.2001771Z * [new branch] gh/williamwen42/307/head -> origin/gh/williamwen42/307/head 2025-11-03T16:15:22.2003409Z * [new branch] gh/williamwen42/307/orig -> origin/gh/williamwen42/307/orig 2025-11-03T16:15:22.2003717Z * [new branch] gh/williamwen42/308/base -> origin/gh/williamwen42/308/base 2025-11-03T16:15:22.2003881Z * [new branch] gh/williamwen42/308/head -> origin/gh/williamwen42/308/head 2025-11-03T16:15:22.2006185Z * [new branch] gh/williamwen42/308/orig -> origin/gh/williamwen42/308/orig 2025-11-03T16:15:22.2006515Z * [new branch] gh/williamwen42/309/base -> origin/gh/williamwen42/309/base 2025-11-03T16:15:22.2006751Z * [new branch] gh/williamwen42/309/head -> origin/gh/williamwen42/309/head 2025-11-03T16:15:22.2006931Z * [new branch] gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig 2025-11-03T16:15:22.2010902Z * [new branch] gh/williamwen42/310/base -> origin/gh/williamwen42/310/base 2025-11-03T16:15:22.2011385Z * [new branch] gh/williamwen42/310/head -> origin/gh/williamwen42/310/head 2025-11-03T16:15:22.2011611Z * [new branch] gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig 2025-11-03T16:15:22.2011769Z * [new branch] gh/williamwen42/311/base -> origin/gh/williamwen42/311/base 2025-11-03T16:15:22.2012007Z * [new branch] gh/williamwen42/311/head -> origin/gh/williamwen42/311/head 2025-11-03T16:15:22.2012696Z * [new branch] gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig 2025-11-03T16:15:22.2016551Z * [new branch] gh/williamwen42/312/base -> origin/gh/williamwen42/312/base 2025-11-03T16:15:22.2016738Z * [new branch] gh/williamwen42/312/head -> origin/gh/williamwen42/312/head 2025-11-03T16:15:22.2016876Z * [new branch] gh/williamwen42/312/orig -> origin/gh/williamwen42/312/orig 2025-11-03T16:15:22.2017015Z * [new branch] gh/williamwen42/313/base -> origin/gh/williamwen42/313/base 2025-11-03T16:15:22.2017156Z * [new branch] gh/williamwen42/313/head -> origin/gh/williamwen42/313/head 2025-11-03T16:15:22.2017329Z * [new branch] gh/williamwen42/313/orig -> origin/gh/williamwen42/313/orig 2025-11-03T16:15:22.2018874Z * [new branch] gh/williamwen42/314/base -> origin/gh/williamwen42/314/base 2025-11-03T16:15:22.2019205Z * [new branch] gh/williamwen42/314/head -> origin/gh/williamwen42/314/head 2025-11-03T16:15:22.2019437Z * [new branch] gh/williamwen42/314/orig -> origin/gh/williamwen42/314/orig 2025-11-03T16:15:22.2021509Z * [new branch] gh/williamwen42/315/base -> origin/gh/williamwen42/315/base 2025-11-03T16:15:22.2022006Z * [new branch] gh/williamwen42/315/head -> origin/gh/williamwen42/315/head 2025-11-03T16:15:22.2022290Z * [new branch] gh/williamwen42/315/orig -> origin/gh/williamwen42/315/orig 2025-11-03T16:15:22.2022515Z * [new branch] gh/williamwen42/316/base -> origin/gh/williamwen42/316/base 2025-11-03T16:15:22.2022903Z * [new branch] gh/williamwen42/316/head -> origin/gh/williamwen42/316/head 2025-11-03T16:15:22.2023802Z * [new branch] gh/williamwen42/316/orig -> origin/gh/williamwen42/316/orig 2025-11-03T16:15:22.2026849Z * [new branch] gh/williamwen42/317/base -> origin/gh/williamwen42/317/base 2025-11-03T16:15:22.2027179Z * [new branch] gh/williamwen42/317/head -> origin/gh/williamwen42/317/head 2025-11-03T16:15:22.2027417Z * [new branch] gh/williamwen42/317/orig -> origin/gh/williamwen42/317/orig 2025-11-03T16:15:22.2027590Z * [new branch] gh/williamwen42/318/base -> origin/gh/williamwen42/318/base 2025-11-03T16:15:22.2027830Z * [new branch] gh/williamwen42/318/head -> origin/gh/williamwen42/318/head 2025-11-03T16:15:22.2027985Z * [new branch] gh/williamwen42/318/orig -> origin/gh/williamwen42/318/orig 2025-11-03T16:15:22.2029011Z * [new branch] gh/williamwen42/319/base -> origin/gh/williamwen42/319/base 2025-11-03T16:15:22.2029713Z * [new branch] gh/williamwen42/319/head -> origin/gh/williamwen42/319/head 2025-11-03T16:15:22.2029916Z * [new branch] gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig 2025-11-03T16:15:22.2031162Z * [new branch] gh/williamwen42/320/base -> origin/gh/williamwen42/320/base 2025-11-03T16:15:22.2031451Z * [new branch] gh/williamwen42/320/head -> origin/gh/williamwen42/320/head 2025-11-03T16:15:22.2032291Z * [new branch] gh/williamwen42/320/orig -> origin/gh/williamwen42/320/orig 2025-11-03T16:15:22.2033232Z * [new branch] gh/williamwen42/321/base -> origin/gh/williamwen42/321/base 2025-11-03T16:15:22.2033388Z * [new branch] gh/williamwen42/321/head -> origin/gh/williamwen42/321/head 2025-11-03T16:15:22.2034208Z * [new branch] gh/williamwen42/321/orig -> origin/gh/williamwen42/321/orig 2025-11-03T16:15:22.2038856Z * [new branch] gh/williamwen42/322/base -> origin/gh/williamwen42/322/base 2025-11-03T16:15:22.2039184Z * [new branch] gh/williamwen42/322/head -> origin/gh/williamwen42/322/head 2025-11-03T16:15:22.2039427Z * [new branch] gh/williamwen42/322/orig -> origin/gh/williamwen42/322/orig 2025-11-03T16:15:22.2039687Z * [new branch] gh/williamwen42/323/base -> origin/gh/williamwen42/323/base 2025-11-03T16:15:22.2039857Z * [new branch] gh/williamwen42/323/head -> origin/gh/williamwen42/323/head 2025-11-03T16:15:22.2040522Z * [new branch] gh/williamwen42/323/orig -> origin/gh/williamwen42/323/orig 2025-11-03T16:15:22.2040703Z * [new branch] gh/williamwen42/324/base -> origin/gh/williamwen42/324/base 2025-11-03T16:15:22.2040850Z * [new branch] gh/williamwen42/324/head -> origin/gh/williamwen42/324/head 2025-11-03T16:15:22.2041191Z * [new branch] gh/williamwen42/324/orig -> origin/gh/williamwen42/324/orig 2025-11-03T16:15:22.2044614Z * [new branch] gh/williamwen42/325/base -> origin/gh/williamwen42/325/base 2025-11-03T16:15:22.2044936Z * [new branch] gh/williamwen42/325/head -> origin/gh/williamwen42/325/head 2025-11-03T16:15:22.2045174Z * [new branch] gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig 2025-11-03T16:15:22.2045329Z * [new branch] gh/williamwen42/326/base -> origin/gh/williamwen42/326/base 2025-11-03T16:15:22.2045557Z * [new branch] gh/williamwen42/326/head -> origin/gh/williamwen42/326/head 2025-11-03T16:15:22.2045860Z * [new branch] gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig 2025-11-03T16:15:22.2046422Z * [new branch] gh/williamwen42/327/base -> origin/gh/williamwen42/327/base 2025-11-03T16:15:22.2046941Z * [new branch] gh/williamwen42/327/head -> origin/gh/williamwen42/327/head 2025-11-03T16:15:22.2049583Z * [new branch] gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig 2025-11-03T16:15:22.2049900Z * [new branch] gh/williamwen42/328/base -> origin/gh/williamwen42/328/base 2025-11-03T16:15:22.2050122Z * [new branch] gh/williamwen42/328/head -> origin/gh/williamwen42/328/head 2025-11-03T16:15:22.2050271Z * [new branch] gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig 2025-11-03T16:15:22.2050563Z * [new branch] gh/williamwen42/329/base -> origin/gh/williamwen42/329/base 2025-11-03T16:15:22.2051498Z * [new branch] gh/williamwen42/329/head -> origin/gh/williamwen42/329/head 2025-11-03T16:15:22.2051867Z * [new branch] gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig 2025-11-03T16:15:22.2054700Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-11-03T16:15:22.2055026Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-11-03T16:15:22.2055261Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-11-03T16:15:22.2055407Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-11-03T16:15:22.2055698Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-11-03T16:15:22.2056602Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-11-03T16:15:22.2056890Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-11-03T16:15:22.2058672Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-11-03T16:15:22.2058975Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-11-03T16:15:22.2059118Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-11-03T16:15:22.2060050Z * [new branch] gh/xmfan/289/base -> origin/gh/xmfan/289/base 2025-11-03T16:15:22.2060494Z * [new branch] gh/xmfan/289/head -> origin/gh/xmfan/289/head 2025-11-03T16:15:22.2062580Z * [new branch] gh/xmfan/289/orig -> origin/gh/xmfan/289/orig 2025-11-03T16:15:22.2062880Z * [new branch] gh/xmfan/291/base -> origin/gh/xmfan/291/base 2025-11-03T16:15:22.2063030Z * [new branch] gh/xmfan/291/head -> origin/gh/xmfan/291/head 2025-11-03T16:15:22.2063344Z * [new branch] gh/xmfan/291/orig -> origin/gh/xmfan/291/orig 2025-11-03T16:15:22.2067411Z * [new branch] gh/xmfan/292/base -> origin/gh/xmfan/292/base 2025-11-03T16:15:22.2067715Z * [new branch] gh/xmfan/292/head -> origin/gh/xmfan/292/head 2025-11-03T16:15:22.2067917Z * [new branch] gh/xmfan/292/orig -> origin/gh/xmfan/292/orig 2025-11-03T16:15:22.2068067Z * [new branch] gh/xmfan/295/base -> origin/gh/xmfan/295/base 2025-11-03T16:15:22.2068267Z * [new branch] gh/xmfan/295/head -> origin/gh/xmfan/295/head 2025-11-03T16:15:22.2068889Z * [new branch] gh/xmfan/295/orig -> origin/gh/xmfan/295/orig 2025-11-03T16:15:22.2069122Z * [new branch] gh/xmfan/296/base -> origin/gh/xmfan/296/base 2025-11-03T16:15:22.2070440Z * [new branch] gh/xmfan/296/head -> origin/gh/xmfan/296/head 2025-11-03T16:15:22.2070598Z * [new branch] gh/xmfan/296/orig -> origin/gh/xmfan/296/orig 2025-11-03T16:15:22.2072047Z * [new branch] gh/xmfan/297/base -> origin/gh/xmfan/297/base 2025-11-03T16:15:22.2072211Z * [new branch] gh/xmfan/297/head -> origin/gh/xmfan/297/head 2025-11-03T16:15:22.2073272Z * [new branch] gh/xmfan/297/orig -> origin/gh/xmfan/297/orig 2025-11-03T16:15:22.2074237Z * [new branch] gh/xmfan/298/base -> origin/gh/xmfan/298/base 2025-11-03T16:15:22.2074535Z * [new branch] gh/xmfan/298/head -> origin/gh/xmfan/298/head 2025-11-03T16:15:22.2075556Z * [new branch] gh/xmfan/298/orig -> origin/gh/xmfan/298/orig 2025-11-03T16:15:22.2076053Z * [new branch] gh/xmfan/299/base -> origin/gh/xmfan/299/base 2025-11-03T16:15:22.2078431Z * [new branch] gh/xmfan/299/head -> origin/gh/xmfan/299/head 2025-11-03T16:15:22.2078592Z * [new branch] gh/xmfan/299/orig -> origin/gh/xmfan/299/orig 2025-11-03T16:15:22.2078738Z * [new branch] gh/xmfan/300/base -> origin/gh/xmfan/300/base 2025-11-03T16:15:22.2078879Z * [new branch] gh/xmfan/300/head -> origin/gh/xmfan/300/head 2025-11-03T16:15:22.2079385Z * [new branch] gh/xmfan/300/orig -> origin/gh/xmfan/300/orig 2025-11-03T16:15:22.2081675Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-11-03T16:15:22.2081990Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-11-03T16:15:22.2082210Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-11-03T16:15:22.2082449Z * [new branch] gh/xmfan/302/base -> origin/gh/xmfan/302/base 2025-11-03T16:15:22.2084107Z * [new branch] gh/xmfan/302/head -> origin/gh/xmfan/302/head 2025-11-03T16:15:22.2084401Z * [new branch] gh/xmfan/302/orig -> origin/gh/xmfan/302/orig 2025-11-03T16:15:22.2087085Z * [new branch] gh/xmfan/303/base -> origin/gh/xmfan/303/base 2025-11-03T16:15:22.2087402Z * [new branch] gh/xmfan/303/head -> origin/gh/xmfan/303/head 2025-11-03T16:15:22.2087629Z * [new branch] gh/xmfan/303/orig -> origin/gh/xmfan/303/orig 2025-11-03T16:15:22.2089361Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-11-03T16:15:22.2089670Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-11-03T16:15:22.2089893Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-11-03T16:15:22.2090039Z * [new branch] gh/xmfan/305/base -> origin/gh/xmfan/305/base 2025-11-03T16:15:22.2091677Z * [new branch] gh/xmfan/305/head -> origin/gh/xmfan/305/head 2025-11-03T16:15:22.2091985Z * [new branch] gh/xmfan/305/orig -> origin/gh/xmfan/305/orig 2025-11-03T16:15:22.2092226Z * [new branch] gh/xmfan/306/base -> origin/gh/xmfan/306/base 2025-11-03T16:15:22.2092367Z * [new branch] gh/xmfan/306/head -> origin/gh/xmfan/306/head 2025-11-03T16:15:22.2092673Z * [new branch] gh/xmfan/306/orig -> origin/gh/xmfan/306/orig 2025-11-03T16:15:22.2094593Z * [new branch] gh/xmfan/307/base -> origin/gh/xmfan/307/base 2025-11-03T16:15:22.2094751Z * [new branch] gh/xmfan/307/head -> origin/gh/xmfan/307/head 2025-11-03T16:15:22.2094880Z * [new branch] gh/xmfan/307/orig -> origin/gh/xmfan/307/orig 2025-11-03T16:15:22.2098638Z * [new branch] gh/xmfan/308/base -> origin/gh/xmfan/308/base 2025-11-03T16:15:22.2098927Z * [new branch] gh/xmfan/308/head -> origin/gh/xmfan/308/head 2025-11-03T16:15:22.2099077Z * [new branch] gh/xmfan/308/orig -> origin/gh/xmfan/308/orig 2025-11-03T16:15:22.2099450Z * [new branch] gh/xmfan/309/base -> origin/gh/xmfan/309/base 2025-11-03T16:15:22.2100017Z * [new branch] gh/xmfan/309/head -> origin/gh/xmfan/309/head 2025-11-03T16:15:22.2100171Z * [new branch] gh/xmfan/309/orig -> origin/gh/xmfan/309/orig 2025-11-03T16:15:22.2100760Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-11-03T16:15:22.2101261Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-11-03T16:15:22.2103540Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-11-03T16:15:22.2103856Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-11-03T16:15:22.2104088Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-11-03T16:15:22.2104235Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-11-03T16:15:22.2104824Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-11-03T16:15:22.2105351Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-11-03T16:15:22.2106091Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-11-03T16:15:22.2108745Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-11-03T16:15:22.2108915Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-11-03T16:15:22.2109053Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-11-03T16:15:22.2109308Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-11-03T16:15:22.2109532Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-11-03T16:15:22.2110472Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-11-03T16:15:22.2111541Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-11-03T16:15:22.2112365Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-11-03T16:15:22.2112747Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-11-03T16:15:22.2114095Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-11-03T16:15:22.2115054Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-11-03T16:15:22.2115371Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-11-03T16:15:22.2116648Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-11-03T16:15:22.2116784Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-11-03T16:15:22.2119103Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-11-03T16:15:22.2119267Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-11-03T16:15:22.2119417Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-11-03T16:15:22.2119635Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-11-03T16:15:22.2121533Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-11-03T16:15:22.2121852Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-11-03T16:15:22.2122072Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-11-03T16:15:22.2123724Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-11-03T16:15:22.2124047Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-11-03T16:15:22.2124450Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-11-03T16:15:22.2126126Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-11-03T16:15:22.2126460Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-11-03T16:15:22.2126663Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-11-03T16:15:22.2129061Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-11-03T16:15:22.2129372Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-11-03T16:15:22.2129598Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-11-03T16:15:22.2129751Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-11-03T16:15:22.2130096Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-11-03T16:15:22.2134128Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-11-03T16:15:22.2134439Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-11-03T16:15:22.2134678Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-11-03T16:15:22.2134824Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-11-03T16:15:22.2134948Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-11-03T16:15:22.2135197Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-11-03T16:15:22.2135333Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-11-03T16:15:22.2136743Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-11-03T16:15:22.2137028Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-11-03T16:15:22.2138779Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-11-03T16:15:22.2139101Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-11-03T16:15:22.2139570Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-11-03T16:15:22.2139722Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-11-03T16:15:22.2140226Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-11-03T16:15:22.2141135Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-11-03T16:15:22.2144035Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-11-03T16:15:22.2144350Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-11-03T16:15:22.2144588Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-11-03T16:15:22.2144727Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-11-03T16:15:22.2144993Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-11-03T16:15:22.2145907Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-11-03T16:15:22.2148693Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-11-03T16:15:22.2148857Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-11-03T16:15:22.2149003Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-11-03T16:15:22.2149132Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-11-03T16:15:22.2149424Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-11-03T16:15:22.2149889Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-11-03T16:15:22.2151541Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-11-03T16:15:22.2151721Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-11-03T16:15:22.2152318Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-11-03T16:15:22.2153671Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-11-03T16:15:22.2154085Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-11-03T16:15:22.2154997Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-11-03T16:15:22.2161350Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-11-03T16:15:22.2161688Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-11-03T16:15:22.2161924Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-11-03T16:15:22.2162081Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-11-03T16:15:22.2162327Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-11-03T16:15:22.2162568Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-11-03T16:15:22.2163096Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-11-03T16:15:22.2163266Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-11-03T16:15:22.2163393Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-11-03T16:15:22.2163523Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-11-03T16:15:22.2163654Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-11-03T16:15:22.2163779Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-11-03T16:15:22.2168560Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-11-03T16:15:22.2169048Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-11-03T16:15:22.2169268Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-11-03T16:15:22.2169482Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-11-03T16:15:22.2169634Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-11-03T16:15:22.2169759Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-11-03T16:15:22.2170010Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-11-03T16:15:22.2170631Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-11-03T16:15:22.2170795Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-11-03T16:15:22.2170921Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-11-03T16:15:22.2171060Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-11-03T16:15:22.2175611Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-11-03T16:15:22.2175945Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-11-03T16:15:22.2176167Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-11-03T16:15:22.2176370Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-11-03T16:15:22.2176505Z * [new branch] gh/ydwu4/326/base -> origin/gh/ydwu4/326/base 2025-11-03T16:15:22.2176914Z * [new branch] gh/ydwu4/326/head -> origin/gh/ydwu4/326/head 2025-11-03T16:15:22.2177060Z * [new branch] gh/ydwu4/326/orig -> origin/gh/ydwu4/326/orig 2025-11-03T16:15:22.2177514Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-11-03T16:15:22.2177694Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-11-03T16:15:22.2177822Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-11-03T16:15:22.2182970Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-11-03T16:15:22.2183291Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-11-03T16:15:22.2183512Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-11-03T16:15:22.2183671Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-11-03T16:15:22.2183813Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-11-03T16:15:22.2184062Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-11-03T16:15:22.2188399Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-11-03T16:15:22.2188726Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-11-03T16:15:22.2188948Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-11-03T16:15:22.2189088Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-11-03T16:15:22.2189301Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-11-03T16:15:22.2189447Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-11-03T16:15:22.2189656Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-11-03T16:15:22.2189803Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-11-03T16:15:22.2190006Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-11-03T16:15:22.2190650Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-11-03T16:15:22.2190984Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-11-03T16:15:22.2191127Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-11-03T16:15:22.2191254Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-11-03T16:15:22.2191382Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-11-03T16:15:22.2191707Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-11-03T16:15:22.2192122Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-11-03T16:15:22.2193085Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-11-03T16:15:22.2193632Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-11-03T16:15:22.2194410Z * [new branch] gh/ydwu4/336/base -> origin/gh/ydwu4/336/base 2025-11-03T16:15:22.2194817Z * [new branch] gh/ydwu4/336/head -> origin/gh/ydwu4/336/head 2025-11-03T16:15:22.2198335Z * [new branch] gh/ydwu4/336/orig -> origin/gh/ydwu4/336/orig 2025-11-03T16:15:22.2198500Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-11-03T16:15:22.2198626Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-11-03T16:15:22.2198746Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-11-03T16:15:22.2198881Z * [new branch] gh/ydwu4/338/base -> origin/gh/ydwu4/338/base 2025-11-03T16:15:22.2199425Z * [new branch] gh/ydwu4/338/head -> origin/gh/ydwu4/338/head 2025-11-03T16:15:22.2200667Z * [new branch] gh/ydwu4/338/orig -> origin/gh/ydwu4/338/orig 2025-11-03T16:15:22.2200855Z * [new branch] gh/ydwu4/339/base -> origin/gh/ydwu4/339/base 2025-11-03T16:15:22.2204831Z * [new branch] gh/ydwu4/339/head -> origin/gh/ydwu4/339/head 2025-11-03T16:15:22.2205133Z * [new branch] gh/ydwu4/339/orig -> origin/gh/ydwu4/339/orig 2025-11-03T16:15:22.2205347Z * [new branch] gh/ydwu4/340/base -> origin/gh/ydwu4/340/base 2025-11-03T16:15:22.2205487Z * [new branch] gh/ydwu4/340/head -> origin/gh/ydwu4/340/head 2025-11-03T16:15:22.2205697Z * [new branch] gh/ydwu4/340/orig -> origin/gh/ydwu4/340/orig 2025-11-03T16:15:22.2205830Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-11-03T16:15:22.2206284Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-11-03T16:15:22.2210543Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-11-03T16:15:22.2210847Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-11-03T16:15:22.2211093Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-11-03T16:15:22.2211251Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-11-03T16:15:22.2211467Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-11-03T16:15:22.2211814Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-11-03T16:15:22.2213106Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-11-03T16:15:22.2213518Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-11-03T16:15:22.2215872Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-11-03T16:15:22.2216184Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-11-03T16:15:22.2216428Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-11-03T16:15:22.2216764Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-11-03T16:15:22.2220800Z * [new branch] gh/yushangdi/1/base -> origin/gh/yushangdi/1/base 2025-11-03T16:15:22.2221109Z * [new branch] gh/yushangdi/1/head -> origin/gh/yushangdi/1/head 2025-11-03T16:15:22.2221331Z * [new branch] gh/yushangdi/2/base -> origin/gh/yushangdi/2/base 2025-11-03T16:15:22.2221479Z * [new branch] gh/yushangdi/2/head -> origin/gh/yushangdi/2/head 2025-11-03T16:15:22.2221706Z * [new branch] gh/yushangdi/3/base -> origin/gh/yushangdi/3/base 2025-11-03T16:15:22.2221864Z * [new branch] gh/yushangdi/3/head -> origin/gh/yushangdi/3/head 2025-11-03T16:15:22.2222632Z * [new branch] gh/yushangdi/3/orig -> origin/gh/yushangdi/3/orig 2025-11-03T16:15:22.2225949Z * [new branch] gh/yushangdi/4/base -> origin/gh/yushangdi/4/base 2025-11-03T16:15:22.2226273Z * [new branch] gh/yushangdi/4/head -> origin/gh/yushangdi/4/head 2025-11-03T16:15:22.2226489Z * [new branch] gh/yushangdi/4/orig -> origin/gh/yushangdi/4/orig 2025-11-03T16:15:22.2226641Z * [new branch] gh/yushangdi/5/base -> origin/gh/yushangdi/5/base 2025-11-03T16:15:22.2226880Z * [new branch] gh/yushangdi/5/head -> origin/gh/yushangdi/5/head 2025-11-03T16:15:22.2227021Z * [new branch] gh/yushangdi/5/orig -> origin/gh/yushangdi/5/orig 2025-11-03T16:15:22.2227863Z * [new branch] gh/yushangdi/6/base -> origin/gh/yushangdi/6/base 2025-11-03T16:15:22.2228411Z * [new branch] gh/yushangdi/6/head -> origin/gh/yushangdi/6/head 2025-11-03T16:15:22.2228969Z * [new branch] gh/yushangdi/6/orig -> origin/gh/yushangdi/6/orig 2025-11-03T16:15:22.2229999Z * [new branch] gh/yushangdi/7/base -> origin/gh/yushangdi/7/base 2025-11-03T16:15:22.2230929Z * [new branch] gh/yushangdi/7/head -> origin/gh/yushangdi/7/head 2025-11-03T16:15:22.2231343Z * [new branch] gh/yushangdi/7/orig -> origin/gh/yushangdi/7/orig 2025-11-03T16:15:22.2232532Z * [new branch] gh/yushangdi/8/base -> origin/gh/yushangdi/8/base 2025-11-03T16:15:22.2233132Z * [new branch] gh/yushangdi/8/head -> origin/gh/yushangdi/8/head 2025-11-03T16:15:22.2233740Z * [new branch] gh/yushangdi/8/orig -> origin/gh/yushangdi/8/orig 2025-11-03T16:15:22.2237729Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-11-03T16:15:22.2241845Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-11-03T16:15:22.2242167Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-11-03T16:15:22.2242395Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-11-03T16:15:22.2242542Z * [new branch] gh/zhxchen17/36/base -> origin/gh/zhxchen17/36/base 2025-11-03T16:15:22.2242776Z * [new branch] gh/zhxchen17/36/head -> origin/gh/zhxchen17/36/head 2025-11-03T16:15:22.2242922Z * [new branch] gh/zhxchen17/36/orig -> origin/gh/zhxchen17/36/orig 2025-11-03T16:15:22.2243059Z * [new branch] gh/zhxchen17/37/base -> origin/gh/zhxchen17/37/base 2025-11-03T16:15:22.2243195Z * [new branch] gh/zhxchen17/37/head -> origin/gh/zhxchen17/37/head 2025-11-03T16:15:22.2243413Z * [new branch] gh/zhxchen17/37/orig -> origin/gh/zhxchen17/37/orig 2025-11-03T16:15:22.2243547Z * [new branch] gh/zhxchen17/38/base -> origin/gh/zhxchen17/38/base 2025-11-03T16:15:22.2243685Z * [new branch] gh/zhxchen17/38/head -> origin/gh/zhxchen17/38/head 2025-11-03T16:15:22.2243947Z * [new branch] gh/zhxchen17/38/orig -> origin/gh/zhxchen17/38/orig 2025-11-03T16:15:22.2245655Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-11-03T16:15:22.2245959Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-11-03T16:15:22.2246105Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-11-03T16:15:22.2250544Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-11-03T16:15:22.2250701Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-11-03T16:15:22.2250845Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-11-03T16:15:22.2250974Z * [new branch] gh/zou3519/1198/base -> origin/gh/zou3519/1198/base 2025-11-03T16:15:22.2251095Z * [new branch] gh/zou3519/1198/head -> origin/gh/zou3519/1198/head 2025-11-03T16:15:22.2251229Z * [new branch] gh/zou3519/1198/orig -> origin/gh/zou3519/1198/orig 2025-11-03T16:15:22.2252730Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-11-03T16:15:22.2252895Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-11-03T16:15:22.2255120Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-11-03T16:15:22.2255439Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-11-03T16:15:22.2255646Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-11-03T16:15:22.2256094Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-11-03T16:15:22.2257312Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-11-03T16:15:22.2257474Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-11-03T16:15:22.2259199Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-11-03T16:15:22.2259511Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-11-03T16:15:22.2259725Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-11-03T16:15:22.2262714Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-11-03T16:15:22.2263034Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-11-03T16:15:22.2263264Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-11-03T16:15:22.2263445Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-11-03T16:15:22.2263777Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-11-03T16:15:22.2264730Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-11-03T16:15:22.2266006Z * [new branch] gh/zpcore/16/base -> origin/gh/zpcore/16/base 2025-11-03T16:15:22.2266140Z * [new branch] gh/zpcore/16/head -> origin/gh/zpcore/16/head 2025-11-03T16:15:22.2266642Z * [new branch] gh/zpcore/16/orig -> origin/gh/zpcore/16/orig 2025-11-03T16:15:22.2267846Z * [new branch] gh/zpcore/17/base -> origin/gh/zpcore/17/base 2025-11-03T16:15:22.2268313Z * [new branch] gh/zpcore/17/head -> origin/gh/zpcore/17/head 2025-11-03T16:15:22.2269204Z * [new branch] gh/zpcore/17/orig -> origin/gh/zpcore/17/orig 2025-11-03T16:15:22.2270228Z * [new branch] gh/zpcore/18/base -> origin/gh/zpcore/18/base 2025-11-03T16:15:22.2270819Z * [new branch] gh/zpcore/18/head -> origin/gh/zpcore/18/head 2025-11-03T16:15:22.2271391Z * [new branch] gh/zpcore/18/orig -> origin/gh/zpcore/18/orig 2025-11-03T16:15:22.2272352Z * [new branch] gh/zpcore/19/base -> origin/gh/zpcore/19/base 2025-11-03T16:15:22.2272917Z * [new branch] gh/zpcore/19/head -> origin/gh/zpcore/19/head 2025-11-03T16:15:22.2273860Z * [new branch] gh/zpcore/19/orig -> origin/gh/zpcore/19/orig 2025-11-03T16:15:22.2274913Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-11-03T16:15:22.2275307Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-11-03T16:15:22.2276578Z * [new branch] gh/zpcore/20/base -> origin/gh/zpcore/20/base 2025-11-03T16:15:22.2277109Z * [new branch] gh/zpcore/20/head -> origin/gh/zpcore/20/head 2025-11-03T16:15:22.2278090Z * [new branch] gh/zpcore/20/orig -> origin/gh/zpcore/20/orig 2025-11-03T16:15:22.2279455Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-11-03T16:15:22.2279902Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-11-03T16:15:22.2280931Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-11-03T16:15:22.2282036Z * [new branch] gh/zpcore/22/base -> origin/gh/zpcore/22/base 2025-11-03T16:15:22.2282355Z * [new branch] gh/zpcore/22/head -> origin/gh/zpcore/22/head 2025-11-03T16:15:22.2283276Z * [new branch] gh/zpcore/22/orig -> origin/gh/zpcore/22/orig 2025-11-03T16:15:22.2283851Z * [new branch] gh/zpcore/23/base -> origin/gh/zpcore/23/base 2025-11-03T16:15:22.2284969Z * [new branch] gh/zpcore/23/head -> origin/gh/zpcore/23/head 2025-11-03T16:15:22.2285123Z * [new branch] gh/zpcore/23/orig -> origin/gh/zpcore/23/orig 2025-11-03T16:15:22.2286980Z * [new branch] gh/zpcore/24/base -> origin/gh/zpcore/24/base 2025-11-03T16:15:22.2287177Z * [new branch] gh/zpcore/24/head -> origin/gh/zpcore/24/head 2025-11-03T16:15:22.2287320Z * [new branch] gh/zpcore/24/orig -> origin/gh/zpcore/24/orig 2025-11-03T16:15:22.2291540Z * [new branch] gh/zpcore/25/base -> origin/gh/zpcore/25/base 2025-11-03T16:15:22.2291706Z * [new branch] gh/zpcore/25/head -> origin/gh/zpcore/25/head 2025-11-03T16:15:22.2291830Z * [new branch] gh/zpcore/25/orig -> origin/gh/zpcore/25/orig 2025-11-03T16:15:22.2291976Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-11-03T16:15:22.2292120Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-11-03T16:15:22.2292240Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-11-03T16:15:22.2292379Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-11-03T16:15:22.2293416Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-11-03T16:15:22.2294004Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-11-03T16:15:22.2294698Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-11-03T16:15:22.2295039Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-11-03T16:15:22.2297611Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-11-03T16:15:22.2297763Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-11-03T16:15:22.2297902Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-11-03T16:15:22.2298020Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-11-03T16:15:22.2299044Z * [new branch] google-main -> origin/google-main 2025-11-03T16:15:22.2299339Z * [new branch] greencontext -> origin/greencontext 2025-11-03T16:15:22.2301463Z * [new branch] guangyey/config -> origin/guangyey/config 2025-11-03T16:15:22.2301793Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-11-03T16:15:22.2302027Z * [new branch] guangyey/reimport -> origin/guangyey/reimport 2025-11-03T16:15:22.2302176Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-11-03T16:15:22.2304071Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-11-03T16:15:22.2304454Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-11-03T16:15:22.2304859Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-11-03T16:15:22.2305792Z * [new branch] hc_baseline -> origin/hc_baseline 2025-11-03T16:15:22.2306119Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-11-03T16:15:22.2307964Z * [new branch] hhh_rand -> origin/hhh_rand 2025-11-03T16:15:22.2308138Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-11-03T16:15:22.2311952Z * [new branch] huba/debug_mode -> origin/huba/debug_mode 2025-11-03T16:15:22.2312124Z * [new branch] huba/dtensor_equal -> origin/huba/dtensor_equal 2025-11-03T16:15:22.2312245Z * [new branch] huba/f1 -> origin/huba/f1 2025-11-03T16:15:22.2312596Z * [new branch] huba/local_tensor -> origin/huba/local_tensor 2025-11-03T16:15:22.2312781Z * [new branch] increase-asan-build-memory -> origin/increase-asan-build-memory 2025-11-03T16:15:22.2313179Z * [new branch] inductor-perf-increase-timeout -> origin/inductor-perf-increase-timeout 2025-11-03T16:15:22.2313568Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-11-03T16:15:22.2320160Z * [new branch] inlining -> origin/inlining 2025-11-03T16:15:22.2322802Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-11-03T16:15:22.2323073Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-11-03T16:15:22.2328367Z * [new branch] install_free_tensors -> origin/install_free_tensors 2025-11-03T16:15:22.2330177Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-11-03T16:15:22.2330465Z * [new branch] issue#58739 -> origin/issue#58739 2025-11-03T16:15:22.2335488Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-11-03T16:15:22.2335806Z * [new branch] jathu/sve -> origin/jathu/sve 2025-11-03T16:15:22.2336062Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-11-03T16:15:22.2336253Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-11-03T16:15:22.2336567Z * [new branch] jeanschmidt/autorevert_tag_cisev -> origin/jeanschmidt/autorevert_tag_cisev 2025-11-03T16:15:22.2336838Z * [new branch] jeanschmidt/autorevert_tag_cisev_2 -> origin/jeanschmidt/autorevert_tag_cisev_2 2025-11-03T16:15:22.2337403Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-11-03T16:15:22.2337614Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-11-03T16:15:22.2337762Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-11-03T16:15:22.2338138Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-11-03T16:15:22.2338293Z * [new branch] justinchu/export-warning -> origin/justinchu/export-warning 2025-11-03T16:15:22.2338441Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-11-03T16:15:22.2338631Z * [new branch] justinchuby/onnx-deprecation-msg -> origin/justinchuby/onnx-deprecation-msg 2025-11-03T16:15:22.2338767Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-11-03T16:15:22.2338890Z * [new branch] kainan_test -> origin/kainan_test 2025-11-03T16:15:22.2339065Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-11-03T16:15:22.2339241Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-11-03T16:15:22.2339383Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-11-03T16:15:22.2339551Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-11-03T16:15:22.2339699Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-11-03T16:15:22.2339826Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-11-03T16:15:22.2339951Z * [new branch] llama4-stable -> origin/llama4-stable 2025-11-03T16:15:22.2340063Z * [new branch] logdetfix -> origin/logdetfix 2025-11-03T16:15:22.2340192Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-11-03T16:15:22.2340386Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-11-03T16:15:22.2340520Z * [new branch] lucaskabela/fix_164814 -> origin/lucaskabela/fix_164814 2025-11-03T16:15:22.2340836Z * [new branch] lucaskabela/fix_164823 -> origin/lucaskabela/fix_164823 2025-11-03T16:15:22.2341094Z * [new branch] lucaskabela/fix_164875 -> origin/lucaskabela/fix_164875 2025-11-03T16:15:22.2341257Z * [new branch] lucaskabela/fix_164876 -> origin/lucaskabela/fix_164876 2025-11-03T16:15:22.2341455Z * [new branch] lucaskabela/fix_error_for_named_tuples -> origin/lucaskabela/fix_error_for_named_tuples 2025-11-03T16:15:22.2341611Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-11-03T16:15:22.2341770Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-11-03T16:15:22.2342076Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-11-03T16:15:22.2342367Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-11-03T16:15:22.2347925Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-11-03T16:15:22.2348337Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-11-03T16:15:22.2348621Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-11-03T16:15:22.2348898Z * [new branch] lucaskabela/type_files_in_variables -> origin/lucaskabela/type_files_in_variables 2025-11-03T16:15:22.2349531Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-11-03T16:15:22.2349734Z * [new branch] lucaskabela/typing_variables -> origin/lucaskabela/typing_variables 2025-11-03T16:15:22.2349969Z * [new branch] lucaskabela/typinng_variables_builtin -> origin/lucaskabela/typinng_variables_builtin 2025-11-03T16:15:22.2350132Z * [new branch] lucaskablea/pickle_einops -> origin/lucaskablea/pickle_einops 2025-11-03T16:15:22.2350391Z * [new branch] main -> origin/main 2025-11-03T16:15:22.2350604Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-11-03T16:15:22.2350740Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-11-03T16:15:22.2350877Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-11-03T16:15:22.2354844Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-11-03T16:15:22.2361334Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-11-03T16:15:22.2361648Z * [new branch] malfet-patch-9 -> origin/malfet-patch-9 2025-11-03T16:15:22.2361914Z * [new branch] malfet/add-3.14-ci -> origin/malfet/add-3.14-ci 2025-11-03T16:15:22.2362089Z * [new branch] malfet/be-green-context -> origin/malfet/be-green-context 2025-11-03T16:15:22.2362469Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-11-03T16:15:22.2363156Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-11-03T16:15:22.2363396Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-11-03T16:15:22.2363601Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-11-03T16:15:22.2363752Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-11-03T16:15:22.2364120Z * [new branch] masnesral/pt2_internal_logging -> origin/masnesral/pt2_internal_logging 2025-11-03T16:15:22.2364283Z * [new branch] mem_profiler_stack_trace -> origin/mem_profiler_stack_trace 2025-11-03T16:15:22.2364424Z * [new branch] memory_profiler_stack -> origin/memory_profiler_stack 2025-11-03T16:15:22.2364586Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-11-03T16:15:22.2364708Z * [new branch] mingw_posix -> origin/mingw_posix 2025-11-03T16:15:22.2364864Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-11-03T16:15:22.2364981Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-11-03T16:15:22.2365109Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-11-03T16:15:22.2365248Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-11-03T16:15:22.2365408Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-11-03T16:15:22.2365556Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-11-03T16:15:22.2365682Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-11-03T16:15:22.2365857Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-11-03T16:15:22.2370227Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-11-03T16:15:22.2370541Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-11-03T16:15:22.2370761Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-11-03T16:15:22.2370912Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-11-03T16:15:22.2371121Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-11-03T16:15:22.2371311Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-11-03T16:15:22.2371857Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-11-03T16:15:22.2372036Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-11-03T16:15:22.2372342Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-11-03T16:15:22.2372527Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-11-03T16:15:22.2372673Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-11-03T16:15:22.2377228Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-11-03T16:15:22.2377575Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-11-03T16:15:22.2377825Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-11-03T16:15:22.2377996Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-11-03T16:15:22.2378135Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-11-03T16:15:22.2378404Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-11-03T16:15:22.2379031Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-11-03T16:15:22.2379203Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-11-03T16:15:22.2379325Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-11-03T16:15:22.2379491Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-11-03T16:15:22.2379628Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-11-03T16:15:22.2379742Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-11-03T16:15:22.2383872Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-11-03T16:15:22.2384222Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-11-03T16:15:22.2384447Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-11-03T16:15:22.2384662Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-11-03T16:15:22.2384811Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-11-03T16:15:22.2385014Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-11-03T16:15:22.2385160Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-11-03T16:15:22.2385801Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-11-03T16:15:22.2386419Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-11-03T16:15:22.2386599Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-11-03T16:15:22.2386730Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-11-03T16:15:22.2386859Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-11-03T16:15:22.2387004Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-11-03T16:15:22.2389797Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-11-03T16:15:22.2389949Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-11-03T16:15:22.2390065Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-11-03T16:15:22.2390186Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-11-03T16:15:22.2390302Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-11-03T16:15:22.2390412Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-11-03T16:15:22.2390551Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-11-03T16:15:22.2391094Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-11-03T16:15:22.2391767Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-11-03T16:15:22.2392346Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-11-03T16:15:22.2392781Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-11-03T16:15:22.2393759Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-11-03T16:15:22.2394193Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-11-03T16:15:22.2395431Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-11-03T16:15:22.2395718Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-11-03T16:15:22.2398502Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-11-03T16:15:22.2398847Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-11-03T16:15:22.2399101Z * [new branch] mlazos/main -> origin/mlazos/main 2025-11-03T16:15:22.2399303Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-11-03T16:15:22.2399543Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-11-03T16:15:22.2400154Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-11-03T16:15:22.2400336Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-11-03T16:15:22.2401914Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-11-03T16:15:22.2402282Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-11-03T16:15:22.2402686Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-11-03T16:15:22.2402912Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-11-03T16:15:22.2403856Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-11-03T16:15:22.2404240Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-11-03T16:15:22.2406552Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-11-03T16:15:22.2406884Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-11-03T16:15:22.2407130Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-11-03T16:15:22.2407292Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-11-03T16:15:22.2407448Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-11-03T16:15:22.2408460Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-11-03T16:15:22.2408698Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-11-03T16:15:22.2411621Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-11-03T16:15:22.2411963Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-11-03T16:15:22.2412206Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-11-03T16:15:22.2412441Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-11-03T16:15:22.2412610Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-11-03T16:15:22.2412845Z * [new branch] mlazos/test -> origin/mlazos/test 2025-11-03T16:15:22.2412977Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-11-03T16:15:22.2413935Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-11-03T16:15:22.2416585Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-11-03T16:15:22.2417154Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-11-03T16:15:22.2417397Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-11-03T16:15:22.2417561Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-11-03T16:15:22.2417693Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-11-03T16:15:22.2418256Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-11-03T16:15:22.2418640Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-11-03T16:15:22.2419974Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-11-03T16:15:22.2420337Z * [new branch] mlazos/user-stream-base -> origin/mlazos/user-stream-base 2025-11-03T16:15:22.2420670Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-11-03T16:15:22.2422939Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-11-03T16:15:22.2423282Z * [new branch] mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2 2025-11-03T16:15:22.2423525Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-11-03T16:15:22.2423891Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-11-03T16:15:22.2424802Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-11-03T16:15:22.2425262Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-11-03T16:15:22.2427601Z * [new branch] module-shim -> origin/module-shim 2025-11-03T16:15:22.2427982Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-11-03T16:15:22.2428270Z * [new branch] move_aws_steps_inside_setup_rocm -> origin/move_aws_steps_inside_setup_rocm 2025-11-03T16:15:22.2428836Z * [new branch] msaroufim-patch-2 -> origin/msaroufim-patch-2 2025-11-03T16:15:22.2429161Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-11-03T16:15:22.2430600Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-11-03T16:15:22.2430841Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-11-03T16:15:22.2431789Z * [new branch] mwizak/restrict-test-mm-backend -> origin/mwizak/restrict-test-mm-backend 2025-11-03T16:15:22.2432033Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-11-03T16:15:22.2433122Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-11-03T16:15:22.2433441Z * [new branch] new-codegen -> origin/new-codegen 2025-11-03T16:15:22.2434447Z * [new branch] newtest-base -> origin/newtest-base 2025-11-03T16:15:22.2437546Z * [new branch] ngimel/discont_allgather -> origin/ngimel/discont_allgather 2025-11-03T16:15:22.2437717Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-11-03T16:15:22.2437846Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-11-03T16:15:22.2437973Z * [new branch] ngimel/nDim_fix -> origin/ngimel/nDim_fix 2025-11-03T16:15:22.2438093Z * [new branch] nightly -> origin/nightly 2025-11-03T16:15:22.2438530Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-11-03T16:15:22.2440377Z * [new branch] nikitaved/addmm_epilogue_fusions -> origin/nikitaved/addmm_epilogue_fusions 2025-11-03T16:15:22.2440609Z * [new branch] nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias 2025-11-03T16:15:22.2440990Z * [new branch] nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor 2025-11-03T16:15:22.2441383Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-11-03T16:15:22.2445852Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-11-03T16:15:22.2446167Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-11-03T16:15:22.2446449Z * [new branch] nmacchioni-40p-machine-balance -> origin/nmacchioni-40p-machine-balance 2025-11-03T16:15:22.2446776Z * [new branch] nmacchioni-60p-machine-balance -> origin/nmacchioni-60p-machine-balance 2025-11-03T16:15:22.2447124Z * [new branch] nmacchioni-padmm-no-compute-bound-check -> origin/nmacchioni-padmm-no-compute-bound-check 2025-11-03T16:15:22.2447732Z * [new branch] nmacchioni-patch-1 -> origin/nmacchioni-patch-1 2025-11-03T16:15:22.2447914Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-11-03T16:15:22.2448040Z * [new branch] nofun-hack -> origin/nofun-hack 2025-11-03T16:15:22.2448337Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-11-03T16:15:22.2448787Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-11-03T16:15:22.2452928Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-11-03T16:15:22.2453245Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-11-03T16:15:22.2453601Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-11-03T16:15:22.2453851Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-11-03T16:15:22.2454007Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-11-03T16:15:22.2454136Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-11-03T16:15:22.2454795Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-11-03T16:15:22.2455205Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-11-03T16:15:22.2456364Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-11-03T16:15:22.2456610Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-11-03T16:15:22.2457134Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-11-03T16:15:22.2459391Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-11-03T16:15:22.2459703Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-11-03T16:15:22.2459931Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-11-03T16:15:22.2460071Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-11-03T16:15:22.2460865Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-11-03T16:15:22.2463810Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-11-03T16:15:22.2464114Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-11-03T16:15:22.2464400Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-11-03T16:15:22.2464563Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-11-03T16:15:22.2465904Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-11-03T16:15:22.2466185Z * [new branch] padded-tensor -> origin/padded-tensor 2025-11-03T16:15:22.2466679Z * [new branch] pca2 -> origin/pca2 2025-11-03T16:15:22.2468104Z * [new branch] per_channel_backup -> origin/per_channel_backup 2025-11-03T16:15:22.2468245Z * [new branch] perf_ops -> origin/perf_ops 2025-11-03T16:15:22.2469770Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-11-03T16:15:22.2469946Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-11-03T16:15:22.2470992Z * [new branch] pianpwk-patch-2 -> origin/pianpwk-patch-2 2025-11-03T16:15:22.2471379Z * [new branch] pianpwk-patch-3 -> origin/pianpwk-patch-3 2025-11-03T16:15:22.2472594Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-11-03T16:15:22.2472804Z * [new branch] pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft 2025-11-03T16:15:22.2473585Z * [new branch] pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys 2025-11-03T16:15:22.2474087Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-11-03T16:15:22.2475368Z * [new branch] pianpwk/anomaly_tb -> origin/pianpwk/anomaly_tb 2025-11-03T16:15:22.2475554Z * [new branch] pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate 2025-11-03T16:15:22.2477786Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-11-03T16:15:22.2478135Z * [new branch] pianpwk/base_view_shape_key -> origin/pianpwk/base_view_shape_key 2025-11-03T16:15:22.2478549Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-11-03T16:15:22.2478854Z * [new branch] pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces 2025-11-03T16:15:22.2479544Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-11-03T16:15:22.2479956Z * [new branch] pianpwk/debug_mode_hooks -> origin/pianpwk/debug_mode_hooks 2025-11-03T16:15:22.2480434Z * [new branch] pianpwk/debug_mode_inductor -> origin/pianpwk/debug_mode_inductor 2025-11-03T16:15:22.2482799Z * [new branch] pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor 2025-11-03T16:15:22.2483143Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-11-03T16:15:22.2483447Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-11-03T16:15:22.2483731Z * [new branch] pianpwk/debugmode_show_ids -> origin/pianpwk/debugmode_show_ids 2025-11-03T16:15:22.2484037Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-11-03T16:15:22.2484626Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-11-03T16:15:22.2485154Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-11-03T16:15:22.2488470Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-11-03T16:15:22.2488807Z * [new branch] pianpwk/event_list_tree -> origin/pianpwk/event_list_tree 2025-11-03T16:15:22.2489059Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-11-03T16:15:22.2489303Z * [new branch] pianpwk/fx_export_annotate -> origin/pianpwk/fx_export_annotate 2025-11-03T16:15:22.2489554Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-11-03T16:15:22.2490169Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-11-03T16:15:22.2490542Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-11-03T16:15:22.2490746Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-11-03T16:15:22.2490931Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-11-03T16:15:22.2492423Z * [new branch] pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate 2025-11-03T16:15:22.2492787Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-11-03T16:15:22.2493118Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-11-03T16:15:22.2495157Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-11-03T16:15:22.2495500Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-11-03T16:15:22.2495790Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-11-03T16:15:22.2496022Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-11-03T16:15:22.2496261Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-11-03T16:15:22.2496705Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-11-03T16:15:22.2501265Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-11-03T16:15:22.2501590Z * [new branch] piz/fix_static_shard_method -> origin/piz/fix_static_shard_method 2025-11-03T16:15:22.2501936Z * [new branch] pool-separate -> origin/pool-separate 2025-11-03T16:15:22.2502187Z * [new branch] pr-156087 -> origin/pr-156087 2025-11-03T16:15:22.2502323Z * [new branch] pr/131860 -> origin/pr/131860 2025-11-03T16:15:22.2502512Z * [new branch] pr165329 -> origin/pr165329 2025-11-03T16:15:22.2503098Z * [new branch] pr165330 -> origin/pr165330 2025-11-03T16:15:22.2503233Z * [new branch] pr165564 -> origin/pr165564 2025-11-03T16:15:22.2503345Z * [new branch] pr165636 -> origin/pr165636 2025-11-03T16:15:22.2504482Z * [new branch] pr165637 -> origin/pr165637 2025-11-03T16:15:22.2504935Z * [new branch] pr165638 -> origin/pr165638 2025-11-03T16:15:22.2505288Z * [new branch] pr165639 -> origin/pr165639 2025-11-03T16:15:22.2506748Z * [new branch] pr165640 -> origin/pr165640 2025-11-03T16:15:22.2507036Z * [new branch] pr165642 -> origin/pr165642 2025-11-03T16:15:22.2507239Z * [new branch] pr165994 -> origin/pr165994 2025-11-03T16:15:22.2509086Z * [new branch] pr166102 -> origin/pr166102 2025-11-03T16:15:22.2509223Z * [new branch] pr166103 -> origin/pr166103 2025-11-03T16:15:22.2509328Z * [new branch] pr166149 -> origin/pr166149 2025-11-03T16:15:22.2511383Z * [new branch] pr166182 -> origin/pr166182 2025-11-03T16:15:22.2511549Z * [new branch] predispatch_to -> origin/predispatch_to 2025-11-03T16:15:22.2511771Z * [new branch] prepare-perf-baseline-number-2.8 -> origin/prepare-perf-baseline-number-2.8 2025-11-03T16:15:22.2511956Z * [new branch] prepare-perf-number-2.9 -> origin/prepare-perf-number-2.9 2025-11-03T16:15:22.2512874Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-11-03T16:15:22.2514350Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-11-03T16:15:22.2517450Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-11-03T16:15:22.2521763Z * [new branch] quint-bits -> origin/quint-bits 2025-11-03T16:15:22.2522112Z * [new branch] reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn 2025-11-03T16:15:22.2522377Z * [new branch] reland-fx-annotate -> origin/reland-fx-annotate 2025-11-03T16:15:22.2522581Z * [new branch] release/1.10 -> origin/release/1.10 2025-11-03T16:15:22.2522785Z * [new branch] release/1.11 -> origin/release/1.11 2025-11-03T16:15:22.2522928Z * [new branch] release/1.12 -> origin/release/1.12 2025-11-03T16:15:22.2523042Z * [new branch] release/1.13 -> origin/release/1.13 2025-11-03T16:15:22.2523276Z * [new branch] release/1.4 -> origin/release/1.4 2025-11-03T16:15:22.2523403Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-11-03T16:15:22.2523511Z * [new branch] release/1.5 -> origin/release/1.5 2025-11-03T16:15:22.2523622Z * [new branch] release/1.6 -> origin/release/1.6 2025-11-03T16:15:22.2523742Z * [new branch] release/1.7 -> origin/release/1.7 2025-11-03T16:15:22.2528714Z * [new branch] release/1.8 -> origin/release/1.8 2025-11-03T16:15:22.2529008Z * [new branch] release/1.9 -> origin/release/1.9 2025-11-03T16:15:22.2529428Z * [new branch] release/2.0 -> origin/release/2.0 2025-11-03T16:15:22.2529682Z * [new branch] release/2.1 -> origin/release/2.1 2025-11-03T16:15:22.2529806Z * [new branch] release/2.2 -> origin/release/2.2 2025-11-03T16:15:22.2529927Z * [new branch] release/2.3 -> origin/release/2.3 2025-11-03T16:15:22.2530034Z * [new branch] release/2.4 -> origin/release/2.4 2025-11-03T16:15:22.2530153Z * [new branch] release/2.5 -> origin/release/2.5 2025-11-03T16:15:22.2530260Z * [new branch] release/2.6 -> origin/release/2.6 2025-11-03T16:15:22.2530376Z * [new branch] release/2.7 -> origin/release/2.7 2025-11-03T16:15:22.2536213Z * [new branch] release/2.8 -> origin/release/2.8 2025-11-03T16:15:22.2536513Z * [new branch] release/2.9 -> origin/release/2.9 2025-11-03T16:15:22.2536726Z * [new branch] release_notes -> origin/release_notes 2025-11-03T16:15:22.2536952Z * [new branch] remove-meta-files -> origin/remove-meta-files 2025-11-03T16:15:22.2537115Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-11-03T16:15:22.2537435Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-11-03T16:15:22.2538115Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-11-03T16:15:22.2538348Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-11-03T16:15:22.2538560Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-11-03T16:15:22.2538808Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-11-03T16:15:22.2539006Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-11-03T16:15:22.2543438Z * [new branch] revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head 2025-11-03T16:15:22.2544051Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-11-03T16:15:22.2544466Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-11-03T16:15:22.2544655Z * [new branch] revert-165144-gh/fadara01/2/head -> origin/revert-165144-gh/fadara01/2/head 2025-11-03T16:15:22.2544787Z * [new branch] revert-cpp -> origin/revert-cpp 2025-11-03T16:15:22.2544921Z * [new branch] revert-failed -> origin/revert-failed 2025-11-03T16:15:22.2545101Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-11-03T16:15:22.2545228Z * [new branch] rms_norm_patch -> origin/rms_norm_patch 2025-11-03T16:15:22.2545493Z * [new branch] ruisi/aot_eager_pass -> origin/ruisi/aot_eager_pass 2025-11-03T16:15:22.2547020Z * [new branch] ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass 2025-11-03T16:15:22.2547196Z * [new branch] ruisi/placement_trace -> origin/ruisi/placement_trace 2025-11-03T16:15:22.2547773Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-11-03T16:15:22.2548275Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-11-03T16:15:22.2549560Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-11-03T16:15:22.2550317Z * [new branch] rzou/njt -> origin/rzou/njt 2025-11-03T16:15:22.2550504Z * [new branch] rzou/pca -> origin/rzou/pca 2025-11-03T16:15:22.2551171Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-11-03T16:15:22.2551796Z * [new branch] samplevllm -> origin/samplevllm 2025-11-03T16:15:22.2553204Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-11-03T16:15:22.2553509Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-11-03T16:15:22.2554890Z * [new branch] sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain 2025-11-03T16:15:22.2555147Z * [new branch] save -> origin/save 2025-11-03T16:15:22.2555650Z * [new branch] save_github_env_rocm -> origin/save_github_env_rocm 2025-11-03T16:15:22.2557767Z * [new branch] sdpa-bs-zero -> origin/sdpa-bs-zero 2025-11-03T16:15:22.2558058Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-11-03T16:15:22.2558291Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-11-03T16:15:22.2559726Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-11-03T16:15:22.2559933Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-11-03T16:15:22.2561463Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-11-03T16:15:22.2561947Z * [new branch] soulitzer/reland-codev-grad-dtype -> origin/soulitzer/reland-codev-grad-dtype 2025-11-03T16:15:22.2562407Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-11-03T16:15:22.2563705Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-11-03T16:15:22.2564276Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-11-03T16:15:22.2564448Z * [new branch] suo -> origin/suo 2025-11-03T16:15:22.2565940Z * [new branch] sve-poc -> origin/sve-poc 2025-11-03T16:15:22.2566227Z * [new branch] switch-bn -> origin/switch-bn 2025-11-03T16:15:22.2566548Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-11-03T16:15:22.2568267Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-11-03T16:15:22.2568605Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-11-03T16:15:22.2568881Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-11-03T16:15:22.2569355Z * [new branch] test-old -> origin/test-old 2025-11-03T16:15:22.2570748Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-11-03T16:15:22.2570931Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-11-03T16:15:22.2573579Z * [new branch] test/inductor -> origin/test/inductor 2025-11-03T16:15:22.2573924Z * [new branch] tianren/customOp_autotune -> origin/tianren/customOp_autotune 2025-11-03T16:15:22.2574219Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-11-03T16:15:22.2574428Z * [new branch] tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune 2025-11-03T16:15:22.2574717Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-11-03T16:15:22.2575077Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-11-03T16:15:22.2576613Z * [new branch] tianren/remove_repeate -> origin/tianren/remove_repeate 2025-11-03T16:15:22.2576890Z * [new branch] tianren/test -> origin/tianren/test 2025-11-03T16:15:22.2578085Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-11-03T16:15:22.2578364Z * [new branch] tmp -> origin/tmp 2025-11-03T16:15:22.2579990Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-11-03T16:15:22.2580336Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-11-03T16:15:22.2580920Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-11-03T16:15:22.2581100Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-11-03T16:15:22.2582747Z * [new branch] triton_kernel -> origin/triton_kernel 2025-11-03T16:15:22.2583120Z * [new branch] trunk-tagging-multi-commits -> origin/trunk-tagging-multi-commits 2025-11-03T16:15:22.2583380Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-11-03T16:15:22.2585211Z * [new branch] type_dec -> origin/type_dec 2025-11-03T16:15:22.2585565Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-11-03T16:15:22.2585915Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-11-03T16:15:22.2586341Z * [new branch] update-audio-commit-hash/18392707270-1874-1 -> origin/update-audio-commit-hash/18392707270-1874-1 2025-11-03T16:15:22.2587510Z * [new branch] update-audio-commit-hash/18513819375-1881-1 -> origin/update-audio-commit-hash/18513819375-1881-1 2025-11-03T16:15:22.2588091Z * [new branch] update-audio-commit-hash/18607927372-1885-1 -> origin/update-audio-commit-hash/18607927372-1885-1 2025-11-03T16:15:22.2588560Z * [new branch] update-audio-commit-hash/18925946320-1900-1 -> origin/update-audio-commit-hash/18925946320-1900-1 2025-11-03T16:15:22.2591669Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-11-03T16:15:22.2591920Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-11-03T16:15:22.2592141Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-11-03T16:15:22.2592351Z * [new branch] update-vision-commit-hash/18361653903-1869-1 -> origin/update-vision-commit-hash/18361653903-1869-1 2025-11-03T16:15:22.2592735Z * [new branch] update-vision-commit-hash/18513819375-1881-1 -> origin/update-vision-commit-hash/18513819375-1881-1 2025-11-03T16:15:22.2593100Z * [new branch] update-vision-commit-hash/18701484675-1891-1 -> origin/update-vision-commit-hash/18701484675-1891-1 2025-11-03T16:15:22.2594005Z * [new branch] update-vision-commit-hash/18860003792-1897-1 -> origin/update-vision-commit-hash/18860003792-1897-1 2025-11-03T16:15:22.2594804Z * [new branch] update-vision-commit-hash/18988459983-1903-1 -> origin/update-vision-commit-hash/18988459983-1903-1 2025-11-03T16:15:22.2598005Z * [new branch] update-vllm-commit-hash/18236802781-1857-1 -> origin/update-vllm-commit-hash/18236802781-1857-1 2025-11-03T16:15:22.2598406Z * [new branch] update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1 2025-11-03T16:15:22.2598737Z * [new branch] update-xla-commit-hash/18273597034-206-1 -> origin/update-xla-commit-hash/18273597034-206-1 2025-11-03T16:15:22.2599199Z * [new branch] update-xla-commit-hash/18458620648-207-1 -> origin/update-xla-commit-hash/18458620648-207-1 2025-11-03T16:15:22.2599412Z * [new branch] update-xla-commit-hash/18645596657-208-1 -> origin/update-xla-commit-hash/18645596657-208-1 2025-11-03T16:15:22.2599619Z * [new branch] update-xla-commit-hash/19027297646-210-1 -> origin/update-xla-commit-hash/19027297646-210-1 2025-11-03T16:15:22.2599828Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-11-03T16:15:22.2600099Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-11-03T16:15:22.2601327Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-11-03T16:15:22.2601660Z * [new branch] update_slow_tests_1759736444 -> origin/update_slow_tests_1759736444 2025-11-03T16:15:22.2601880Z * [new branch] update_slow_tests_1760341258 -> origin/update_slow_tests_1760341258 2025-11-03T16:15:22.2603486Z * [new branch] update_slow_tests_1760946059 -> origin/update_slow_tests_1760946059 2025-11-03T16:15:22.2603825Z * [new branch] update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677 2025-11-03T16:15:22.2604099Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-11-03T16:15:22.2604488Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-11-03T16:15:22.2605737Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-11-03T16:15:22.2605909Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-11-03T16:15:22.2608555Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-11-03T16:15:22.2608854Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-11-03T16:15:22.2609062Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-11-03T16:15:22.2609196Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-11-03T16:15:22.2609599Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-11-03T16:15:22.2613726Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-11-03T16:15:22.2614075Z * [new branch] validate_fn -> origin/validate_fn 2025-11-03T16:15:22.2614207Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-11-03T16:15:22.2614333Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-11-03T16:15:22.2614450Z * [new branch] varlen-api -> origin/varlen-api 2025-11-03T16:15:22.2614775Z * [new branch] varlen-api-backup -> origin/varlen-api-backup 2025-11-03T16:15:22.2615011Z * [new branch] viable/strict -> origin/viable/strict 2025-11-03T16:15:22.2619324Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-11-03T16:15:22.2619636Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-11-03T16:15:22.2619831Z * [new branch] vllmpin -> origin/vllmpin 2025-11-03T16:15:22.2619982Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-11-03T16:15:22.2620175Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-11-03T16:15:22.2620303Z * [new branch] whc/uneven -> origin/whc/uneven 2025-11-03T16:15:22.2620789Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-11-03T16:15:22.2621701Z * [new branch] win_12_8_build -> origin/win_12_8_build 2025-11-03T16:15:22.2622030Z * [new branch] win_ci -> origin/win_ci 2025-11-03T16:15:22.2625242Z * [new branch] win_warnings -> origin/win_warnings 2025-11-03T16:15:22.2625763Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-11-03T16:15:22.2626033Z * [new branch] windows_mmap -> origin/windows_mmap 2025-11-03T16:15:22.2626286Z * [new branch] xmfan-war -> origin/xmfan-war 2025-11-03T16:15:22.2626420Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-11-03T16:15:22.2627078Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-11-03T16:15:22.2627388Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-11-03T16:15:22.2627633Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-11-03T16:15:22.2628697Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-11-03T16:15:22.2629048Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-11-03T16:15:22.2629500Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-11-03T16:15:22.2632293Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-11-03T16:15:22.2632471Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-11-03T16:15:22.2632601Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-11-03T16:15:22.2632737Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-11-03T16:15:22.2633174Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-11-03T16:15:22.2633586Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-11-03T16:15:22.2634534Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-11-03T16:15:22.2637253Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-11-03T16:15:22.2637414Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-11-03T16:15:22.2637556Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-11-03T16:15:22.2637865Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-11-03T16:15:22.2638036Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-11-03T16:15:22.2639494Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-11-03T16:15:22.2639788Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-11-03T16:15:22.2639939Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-11-03T16:15:22.2641298Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-11-03T16:15:22.2641676Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-11-03T16:15:22.2642062Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-11-03T16:15:22.2642543Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-11-03T16:15:22.2644194Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-11-03T16:15:22.2644501Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-11-03T16:15:22.2644712Z * [new branch] xmfan/test -> origin/xmfan/test 2025-11-03T16:15:22.2647025Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-11-03T16:15:22.2647357Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-11-03T16:15:22.2647851Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-11-03T16:15:22.2648119Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-11-03T16:15:22.2648740Z * [new branch] yiming/add_timm_models -> origin/yiming/add_timm_models 2025-11-03T16:15:22.2648897Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-11-03T16:15:22.2649322Z * [new branch] yiming/fix_aot_joint_graph_capture_test -> origin/yiming/fix_aot_joint_graph_capture_test 2025-11-03T16:15:22.2650549Z * [new branch] yiming/flex_attention_export_with_kwargs -> origin/yiming/flex_attention_export_with_kwargs 2025-11-03T16:15:22.2650790Z * [new branch] yiming/improve_custom_op_stack_trace -> origin/yiming/improve_custom_op_stack_trace 2025-11-03T16:15:22.2654491Z * [new branch] yiming/improve_sharding_error_msg -> origin/yiming/improve_sharding_error_msg 2025-11-03T16:15:22.2654862Z * [new branch] yiming/moe_dtensor_region_annotation -> origin/yiming/moe_dtensor_region_annotation 2025-11-03T16:15:22.2655159Z * [new branch] yiming/precompile_benchmark -> origin/yiming/precompile_benchmark 2025-11-03T16:15:22.2655511Z * [new branch] yiming/remove_blockmask_pytree_global_registration -> origin/yiming/remove_blockmask_pytree_global_registration 2025-11-03T16:15:22.2655776Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-11-03T16:15:22.2656399Z * [new branch] yuxin/fix-save-memleak -> origin/yuxin/fix-save-memleak 2025-11-03T16:15:22.2656557Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-11-03T16:15:22.2656873Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-11-03T16:15:22.2657205Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-11-03T16:15:22.2658009Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-11-03T16:15:22.2658433Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-11-03T16:15:22.2660643Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-11-03T16:15:22.2660934Z * [new branch] zb2p -> origin/zb2p 2025-11-03T16:15:22.2661211Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-11-03T16:15:22.2661633Z * [new branch] zhxchen17/aot_compile_fix_load_guard_manager -> origin/zhxchen17/aot_compile_fix_load_guard_manager 2025-11-03T16:15:22.2663059Z * [new branch] zhxchen17/ci/vllm_pin -> origin/zhxchen17/ci/vllm_pin 2025-11-03T16:15:22.2663437Z * [new branch] zhxchen17/precompile/closure_serde -> origin/zhxchen17/precompile/closure_serde 2025-11-03T16:15:22.2663984Z * [new branch] zhxchen17/precompile/default_args -> origin/zhxchen17/precompile/default_args 2025-11-03T16:15:22.2668204Z * [new branch] zhxchen17/precompile/export -> origin/zhxchen17/precompile/export 2025-11-03T16:15:22.2668562Z * [new branch] zhxchen17/precompile/export_gm -> origin/zhxchen17/precompile/export_gm 2025-11-03T16:15:22.2668884Z * [new branch] zhxchen17/precompile_vllm_fix_0 -> origin/zhxchen17/precompile_vllm_fix_0 2025-11-03T16:15:22.2669124Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-11-03T16:15:22.2669733Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-11-03T16:15:22.2669897Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-11-03T16:15:22.2670050Z * [new branch] zxiiro/c7i-docs-build -> origin/zxiiro/c7i-docs-build 2025-11-03T16:15:22.2670236Z * [new branch] zxiiro/c7i-linux-4xlarge -> origin/zxiiro/c7i-linux-4xlarge 2025-11-03T16:15:22.2671823Z * [new branch] zxiiro/c7i-linux-build-yaml -> origin/zxiiro/c7i-linux-build-yaml 2025-11-03T16:15:22.2672006Z * [new branch] zxiiro/c7i-pull -> origin/zxiiro/c7i-pull 2025-11-03T16:15:22.2672433Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-11-03T16:15:22.2673367Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-11-03T16:15:22.2673776Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-11-03T16:15:22.2677075Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-11-03T16:15:22.2677369Z * [new tag] ciflow/b200/115316 -> ciflow/b200/115316 2025-11-03T16:15:22.2677589Z * [new tag] ciflow/b200/157553 -> ciflow/b200/157553 2025-11-03T16:15:22.2677834Z * [new tag] ciflow/b200/157635 -> ciflow/b200/157635 2025-11-03T16:15:22.2677958Z * [new tag] ciflow/b200/160685 -> ciflow/b200/160685 2025-11-03T16:15:22.2678069Z * [new tag] ciflow/b200/161404 -> ciflow/b200/161404 2025-11-03T16:15:22.2678304Z * [new tag] ciflow/b200/163369 -> ciflow/b200/163369 2025-11-03T16:15:22.2678899Z * [new tag] ciflow/b200/164657 -> ciflow/b200/164657 2025-11-03T16:15:22.2679037Z * [new tag] ciflow/b200/164938 -> ciflow/b200/164938 2025-11-03T16:15:22.2679155Z * [new tag] ciflow/b200/165132 -> ciflow/b200/165132 2025-11-03T16:15:22.2679554Z * [new tag] ciflow/b200/165541 -> ciflow/b200/165541 2025-11-03T16:15:22.2680075Z * [new tag] ciflow/b200/165548 -> ciflow/b200/165548 2025-11-03T16:15:22.2680765Z * [new tag] ciflow/b200/165952 -> ciflow/b200/165952 2025-11-03T16:15:22.2681220Z * [new tag] ciflow/b200/166250 -> ciflow/b200/166250 2025-11-03T16:15:22.2683832Z * [new tag] ciflow/b200/166526 -> ciflow/b200/166526 2025-11-03T16:15:22.2684290Z * [new tag] ciflow/b200/166744 -> ciflow/b200/166744 2025-11-03T16:15:22.2684493Z * [new tag] ciflow/b200/166752 -> ciflow/b200/166752 2025-11-03T16:15:22.2684607Z * [new tag] ciflow/b200/166834 -> ciflow/b200/166834 2025-11-03T16:15:22.2684824Z * [new tag] ciflow/binaries/157432 -> ciflow/binaries/157432 2025-11-03T16:15:22.2684961Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-11-03T16:15:22.2685141Z * [new tag] ciflow/binaries/165922 -> ciflow/binaries/165922 2025-11-03T16:15:22.2685901Z * [new tag] ciflow/binaries/166044 -> ciflow/binaries/166044 2025-11-03T16:15:22.2686512Z * [new tag] ciflow/binaries/166621 -> ciflow/binaries/166621 2025-11-03T16:15:22.2686792Z * [new tag] ciflow/binaries/166764 -> ciflow/binaries/166764 2025-11-03T16:15:22.2686954Z * [new tag] ciflow/binaries/166818 -> ciflow/binaries/166818 2025-11-03T16:15:22.2691033Z * [new tag] ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432 2025-11-03T16:15:22.2691353Z * [new tag] ciflow/binaries_wheel/155731 -> ciflow/binaries_wheel/155731 2025-11-03T16:15:22.2691564Z * [new tag] ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432 2025-11-03T16:15:22.2691714Z * [new tag] ciflow/binaries_wheel/166380 -> ciflow/binaries_wheel/166380 2025-11-03T16:15:22.2691930Z * [new tag] ciflow/binaries_wheel/166407 -> ciflow/binaries_wheel/166407 2025-11-03T16:15:22.2692077Z * [new tag] ciflow/binaries_wheel/166829 -> ciflow/binaries_wheel/166829 2025-11-03T16:15:22.2692363Z * [new tag] ciflow/h100-distributed/166367 -> ciflow/h100-distributed/166367 2025-11-03T16:15:22.2692505Z * [new tag] ciflow/h100-symm-mem/151845 -> ciflow/h100-symm-mem/151845 2025-11-03T16:15:22.2692641Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-11-03T16:15:22.2692758Z * [new tag] ciflow/h100-symm-mem/163814 -> ciflow/h100-symm-mem/163814 2025-11-03T16:15:22.2692881Z * [new tag] ciflow/h100-symm-mem/163815 -> ciflow/h100-symm-mem/163815 2025-11-03T16:15:22.2693007Z * [new tag] ciflow/h100-symm-mem/165548 -> ciflow/h100-symm-mem/165548 2025-11-03T16:15:22.2693147Z * [new tag] ciflow/h100-symm-mem/166772 -> ciflow/h100-symm-mem/166772 2025-11-03T16:15:22.2693547Z * [new tag] ciflow/h100-symm-mem/166773 -> ciflow/h100-symm-mem/166773 2025-11-03T16:15:22.2693965Z * [new tag] ciflow/h100-symm-mem/166774 -> ciflow/h100-symm-mem/166774 2025-11-03T16:15:22.2695094Z * [new tag] ciflow/h100/115316 -> ciflow/h100/115316 2025-11-03T16:15:22.2695370Z * [new tag] ciflow/h100/157553 -> ciflow/h100/157553 2025-11-03T16:15:22.2695520Z * [new tag] ciflow/h100/157635 -> ciflow/h100/157635 2025-11-03T16:15:22.2695702Z * [new tag] ciflow/h100/160685 -> ciflow/h100/160685 2025-11-03T16:15:22.2696036Z * [new tag] ciflow/h100/161404 -> ciflow/h100/161404 2025-11-03T16:15:22.2696404Z * [new tag] ciflow/h100/163369 -> ciflow/h100/163369 2025-11-03T16:15:22.2696802Z * [new tag] ciflow/h100/164657 -> ciflow/h100/164657 2025-11-03T16:15:22.2697934Z * [new tag] ciflow/h100/165029 -> ciflow/h100/165029 2025-11-03T16:15:22.2698221Z * [new tag] ciflow/h100/165132 -> ciflow/h100/165132 2025-11-03T16:15:22.2698365Z * [new tag] ciflow/h100/165541 -> ciflow/h100/165541 2025-11-03T16:15:22.2698545Z * [new tag] ciflow/h100/165548 -> ciflow/h100/165548 2025-11-03T16:15:22.2699061Z * [new tag] ciflow/h100/165952 -> ciflow/h100/165952 2025-11-03T16:15:22.2700169Z * [new tag] ciflow/h100/166250 -> ciflow/h100/166250 2025-11-03T16:15:22.2700451Z * [new tag] ciflow/h100/166515 -> ciflow/h100/166515 2025-11-03T16:15:22.2700586Z * [new tag] ciflow/h100/166526 -> ciflow/h100/166526 2025-11-03T16:15:22.2701036Z * [new tag] ciflow/h100/166744 -> ciflow/h100/166744 2025-11-03T16:15:22.2701493Z * [new tag] ciflow/h100/166752 -> ciflow/h100/166752 2025-11-03T16:15:22.2701933Z * [new tag] ciflow/h100/166834 -> ciflow/h100/166834 2025-11-03T16:15:22.2703355Z * [new tag] ciflow/inductor-perf-compare/165029 -> ciflow/inductor-perf-compare/165029 2025-11-03T16:15:22.2703816Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/162053 -> ciflow/inductor-perf-test-nightly-rocm-mi300/162053 2025-11-03T16:15:22.2704197Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/162954 -> ciflow/inductor-perf-test-nightly-x86-zen/162954 2025-11-03T16:15:22.2704569Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/164126 -> ciflow/inductor-perf-test-nightly-x86-zen/164126 2025-11-03T16:15:22.2704898Z * [new tag] ciflow/inductor-perf-test-nightly/162053 -> ciflow/inductor-perf-test-nightly/162053 2025-11-03T16:15:22.2705170Z * [new tag] ciflow/inductor-perf-test-nightly/165029 -> ciflow/inductor-perf-test-nightly/165029 2025-11-03T16:15:22.2705783Z * [new tag] ciflow/inductor-periodic/165029 -> ciflow/inductor-periodic/165029 2025-11-03T16:15:22.2706229Z * [new tag] ciflow/inductor-periodic/166743 -> ciflow/inductor-periodic/166743 2025-11-03T16:15:22.2706765Z * [new tag] ciflow/inductor-rocm/151845 -> ciflow/inductor-rocm/151845 2025-11-03T16:15:22.2707306Z * [new tag] ciflow/inductor-rocm/162053 -> ciflow/inductor-rocm/162053 2025-11-03T16:15:22.2707601Z * [new tag] ciflow/inductor-rocm/166515 -> ciflow/inductor-rocm/166515 2025-11-03T16:15:22.2707748Z * [new tag] ciflow/inductor-rocm/166743 -> ciflow/inductor-rocm/166743 2025-11-03T16:15:22.2708805Z * [new tag] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-11-03T16:15:22.2708951Z * [new tag] ciflow/inductor/148180 -> ciflow/inductor/148180 2025-11-03T16:15:22.2709262Z * [new tag] ciflow/inductor/148294 -> ciflow/inductor/148294 2025-11-03T16:15:22.2709564Z * [new tag] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-11-03T16:15:22.2710021Z * [new tag] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-11-03T16:15:22.2710542Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-11-03T16:15:22.2710894Z * [new tag] ciflow/inductor/151845 -> ciflow/inductor/151845 2025-11-03T16:15:22.2711265Z * [new tag] ciflow/inductor/152624 -> ciflow/inductor/152624 2025-11-03T16:15:22.2711599Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-11-03T16:15:22.2712010Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-11-03T16:15:22.2712424Z * [new tag] ciflow/inductor/159523 -> ciflow/inductor/159523 2025-11-03T16:15:22.2714329Z * [new tag] ciflow/inductor/160174 -> ciflow/inductor/160174 2025-11-03T16:15:22.2714474Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-11-03T16:15:22.2717853Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-11-03T16:15:22.2718122Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-11-03T16:15:22.2718499Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-11-03T16:15:22.2719159Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-11-03T16:15:22.2719554Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-11-03T16:15:22.2720027Z * [new tag] ciflow/inductor/161158 -> ciflow/inductor/161158 2025-11-03T16:15:22.2720702Z * [new tag] ciflow/inductor/161246 -> ciflow/inductor/161246 2025-11-03T16:15:22.2721011Z * [new tag] ciflow/inductor/161404 -> ciflow/inductor/161404 2025-11-03T16:15:22.2721409Z * [new tag] ciflow/inductor/161495 -> ciflow/inductor/161495 2025-11-03T16:15:22.2721863Z * [new tag] ciflow/inductor/161512 -> ciflow/inductor/161512 2025-11-03T16:15:22.2723184Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-11-03T16:15:22.2725798Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-11-03T16:15:22.2726125Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-11-03T16:15:22.2726332Z * [new tag] ciflow/inductor/162053 -> ciflow/inductor/162053 2025-11-03T16:15:22.2726466Z * [new tag] ciflow/inductor/162469 -> ciflow/inductor/162469 2025-11-03T16:15:22.2726660Z * [new tag] ciflow/inductor/162709 -> ciflow/inductor/162709 2025-11-03T16:15:22.2726803Z * [new tag] ciflow/inductor/162795 -> ciflow/inductor/162795 2025-11-03T16:15:22.2727443Z * [new tag] ciflow/inductor/162954 -> ciflow/inductor/162954 2025-11-03T16:15:22.2727589Z * [new tag] ciflow/inductor/163249 -> ciflow/inductor/163249 2025-11-03T16:15:22.2727894Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-11-03T16:15:22.2728020Z * [new tag] ciflow/inductor/163369 -> ciflow/inductor/163369 2025-11-03T16:15:22.2728140Z * [new tag] ciflow/inductor/163490 -> ciflow/inductor/163490 2025-11-03T16:15:22.2728253Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-11-03T16:15:22.2728375Z * [new tag] ciflow/inductor/163714 -> ciflow/inductor/163714 2025-11-03T16:15:22.2728484Z * [new tag] ciflow/inductor/163936 -> ciflow/inductor/163936 2025-11-03T16:15:22.2728600Z * [new tag] ciflow/inductor/164202 -> ciflow/inductor/164202 2025-11-03T16:15:22.2729062Z * [new tag] ciflow/inductor/164384 -> ciflow/inductor/164384 2025-11-03T16:15:22.2729300Z * [new tag] ciflow/inductor/164657 -> ciflow/inductor/164657 2025-11-03T16:15:22.2729717Z * [new tag] ciflow/inductor/164938 -> ciflow/inductor/164938 2025-11-03T16:15:22.2732738Z * [new tag] ciflow/inductor/164979 -> ciflow/inductor/164979 2025-11-03T16:15:22.2732903Z * [new tag] ciflow/inductor/164980 -> ciflow/inductor/164980 2025-11-03T16:15:22.2733023Z * [new tag] ciflow/inductor/165005 -> ciflow/inductor/165005 2025-11-03T16:15:22.2733130Z * [new tag] ciflow/inductor/165029 -> ciflow/inductor/165029 2025-11-03T16:15:22.2733246Z * [new tag] ciflow/inductor/165092 -> ciflow/inductor/165092 2025-11-03T16:15:22.2733354Z * [new tag] ciflow/inductor/165132 -> ciflow/inductor/165132 2025-11-03T16:15:22.2733463Z * [new tag] ciflow/inductor/165197 -> ciflow/inductor/165197 2025-11-03T16:15:22.2733746Z * [new tag] ciflow/inductor/165274 -> ciflow/inductor/165274 2025-11-03T16:15:22.2733894Z * [new tag] ciflow/inductor/165283 -> ciflow/inductor/165283 2025-11-03T16:15:22.2734163Z * [new tag] ciflow/inductor/165284 -> ciflow/inductor/165284 2025-11-03T16:15:22.2734708Z * [new tag] ciflow/inductor/165367 -> ciflow/inductor/165367 2025-11-03T16:15:22.2735708Z * [new tag] ciflow/inductor/165423 -> ciflow/inductor/165423 2025-11-03T16:15:22.2735833Z * [new tag] ciflow/inductor/165487 -> ciflow/inductor/165487 2025-11-03T16:15:22.2736023Z * [new tag] ciflow/inductor/165541 -> ciflow/inductor/165541 2025-11-03T16:15:22.2736436Z * [new tag] ciflow/inductor/165597 -> ciflow/inductor/165597 2025-11-03T16:15:22.2738235Z * [new tag] ciflow/inductor/165686 -> ciflow/inductor/165686 2025-11-03T16:15:22.2738532Z * [new tag] ciflow/inductor/165790 -> ciflow/inductor/165790 2025-11-03T16:15:22.2738779Z * [new tag] ciflow/inductor/165824 -> ciflow/inductor/165824 2025-11-03T16:15:22.2738911Z * [new tag] ciflow/inductor/165856 -> ciflow/inductor/165856 2025-11-03T16:15:22.2739108Z * [new tag] ciflow/inductor/165885 -> ciflow/inductor/165885 2025-11-03T16:15:22.2739238Z * [new tag] ciflow/inductor/165952 -> ciflow/inductor/165952 2025-11-03T16:15:22.2739718Z * [new tag] ciflow/inductor/165953 -> ciflow/inductor/165953 2025-11-03T16:15:22.2740068Z * [new tag] ciflow/inductor/166071 -> ciflow/inductor/166071 2025-11-03T16:15:22.2743014Z * [new tag] ciflow/inductor/166083 -> ciflow/inductor/166083 2025-11-03T16:15:22.2743315Z * [new tag] ciflow/inductor/166170 -> ciflow/inductor/166170 2025-11-03T16:15:22.2743541Z * [new tag] ciflow/inductor/166198 -> ciflow/inductor/166198 2025-11-03T16:15:22.2743823Z * [new tag] ciflow/inductor/166225 -> ciflow/inductor/166225 2025-11-03T16:15:22.2743940Z * [new tag] ciflow/inductor/166236 -> ciflow/inductor/166236 2025-11-03T16:15:22.2744063Z * [new tag] ciflow/inductor/166250 -> ciflow/inductor/166250 2025-11-03T16:15:22.2744327Z * [new tag] ciflow/inductor/166254 -> ciflow/inductor/166254 2025-11-03T16:15:22.2744466Z * [new tag] ciflow/inductor/166276 -> ciflow/inductor/166276 2025-11-03T16:15:22.2744582Z * [new tag] ciflow/inductor/166294 -> ciflow/inductor/166294 2025-11-03T16:15:22.2745066Z * [new tag] ciflow/inductor/166300 -> ciflow/inductor/166300 2025-11-03T16:15:22.2745485Z * [new tag] ciflow/inductor/166320 -> ciflow/inductor/166320 2025-11-03T16:15:22.2745935Z * [new tag] ciflow/inductor/166321 -> ciflow/inductor/166321 2025-11-03T16:15:22.2746363Z * [new tag] ciflow/inductor/166337 -> ciflow/inductor/166337 2025-11-03T16:15:22.2748255Z * [new tag] ciflow/inductor/166341 -> ciflow/inductor/166341 2025-11-03T16:15:22.2748408Z * [new tag] ciflow/inductor/166354 -> ciflow/inductor/166354 2025-11-03T16:15:22.2748541Z * [new tag] ciflow/inductor/166355 -> ciflow/inductor/166355 2025-11-03T16:15:22.2748660Z * [new tag] ciflow/inductor/166361 -> ciflow/inductor/166361 2025-11-03T16:15:22.2748923Z * [new tag] ciflow/inductor/166366 -> ciflow/inductor/166366 2025-11-03T16:15:22.2749065Z * [new tag] ciflow/inductor/166367 -> ciflow/inductor/166367 2025-11-03T16:15:22.2750477Z * [new tag] ciflow/inductor/166370 -> ciflow/inductor/166370 2025-11-03T16:15:22.2750628Z * [new tag] ciflow/inductor/166371 -> ciflow/inductor/166371 2025-11-03T16:15:22.2750761Z * [new tag] ciflow/inductor/166372 -> ciflow/inductor/166372 2025-11-03T16:15:22.2751065Z * [new tag] ciflow/inductor/166377 -> ciflow/inductor/166377 2025-11-03T16:15:22.2751436Z * [new tag] ciflow/inductor/166400 -> ciflow/inductor/166400 2025-11-03T16:15:22.2751948Z * [new tag] ciflow/inductor/166402 -> ciflow/inductor/166402 2025-11-03T16:15:22.2752298Z * [new tag] ciflow/inductor/166417 -> ciflow/inductor/166417 2025-11-03T16:15:22.2752798Z * [new tag] ciflow/inductor/166433 -> ciflow/inductor/166433 2025-11-03T16:15:22.2753546Z * [new tag] ciflow/inductor/166442 -> ciflow/inductor/166442 2025-11-03T16:15:22.2753690Z * [new tag] ciflow/inductor/166467 -> ciflow/inductor/166467 2025-11-03T16:15:22.2756843Z * [new tag] ciflow/inductor/166479 -> ciflow/inductor/166479 2025-11-03T16:15:22.2757145Z * [new tag] ciflow/inductor/166482 -> ciflow/inductor/166482 2025-11-03T16:15:22.2757361Z * [new tag] ciflow/inductor/166483 -> ciflow/inductor/166483 2025-11-03T16:15:22.2757582Z * [new tag] ciflow/inductor/166494 -> ciflow/inductor/166494 2025-11-03T16:15:22.2757833Z * [new tag] ciflow/inductor/166498 -> ciflow/inductor/166498 2025-11-03T16:15:22.2757960Z * [new tag] ciflow/inductor/166499 -> ciflow/inductor/166499 2025-11-03T16:15:22.2758198Z * [new tag] ciflow/inductor/166515 -> ciflow/inductor/166515 2025-11-03T16:15:22.2758340Z * [new tag] ciflow/inductor/166536 -> ciflow/inductor/166536 2025-11-03T16:15:22.2759028Z * [new tag] ciflow/inductor/166540 -> ciflow/inductor/166540 2025-11-03T16:15:22.2759337Z * [new tag] ciflow/inductor/166541 -> ciflow/inductor/166541 2025-11-03T16:15:22.2759541Z * [new tag] ciflow/inductor/166545 -> ciflow/inductor/166545 2025-11-03T16:15:22.2760324Z * [new tag] ciflow/inductor/166581 -> ciflow/inductor/166581 2025-11-03T16:15:22.2760494Z * [new tag] ciflow/inductor/166584 -> ciflow/inductor/166584 2025-11-03T16:15:22.2760784Z * [new tag] ciflow/inductor/166593 -> ciflow/inductor/166593 2025-11-03T16:15:22.2761071Z * [new tag] ciflow/inductor/166600 -> ciflow/inductor/166600 2025-11-03T16:15:22.2761498Z * [new tag] ciflow/inductor/166608 -> ciflow/inductor/166608 2025-11-03T16:15:22.2765879Z * [new tag] ciflow/inductor/166610 -> ciflow/inductor/166610 2025-11-03T16:15:22.2766193Z * [new tag] ciflow/inductor/166629 -> ciflow/inductor/166629 2025-11-03T16:15:22.2766341Z * [new tag] ciflow/inductor/166640 -> ciflow/inductor/166640 2025-11-03T16:15:22.2766559Z * [new tag] ciflow/inductor/166648 -> ciflow/inductor/166648 2025-11-03T16:15:22.2766718Z * [new tag] ciflow/inductor/166658 -> ciflow/inductor/166658 2025-11-03T16:15:22.2766945Z * [new tag] ciflow/inductor/166659 -> ciflow/inductor/166659 2025-11-03T16:15:22.2767542Z * [new tag] ciflow/inductor/166662 -> ciflow/inductor/166662 2025-11-03T16:15:22.2767703Z * [new tag] ciflow/inductor/166663 -> ciflow/inductor/166663 2025-11-03T16:15:22.2767828Z * [new tag] ciflow/inductor/166664 -> ciflow/inductor/166664 2025-11-03T16:15:22.2767976Z * [new tag] ciflow/inductor/166669 -> ciflow/inductor/166669 2025-11-03T16:15:22.2768103Z * [new tag] ciflow/inductor/166673 -> ciflow/inductor/166673 2025-11-03T16:15:22.2768220Z * [new tag] ciflow/inductor/166674 -> ciflow/inductor/166674 2025-11-03T16:15:22.2768517Z * [new tag] ciflow/inductor/166676 -> ciflow/inductor/166676 2025-11-03T16:15:22.2768671Z * [new tag] ciflow/inductor/166679 -> ciflow/inductor/166679 2025-11-03T16:15:22.2769384Z * [new tag] ciflow/inductor/166701 -> ciflow/inductor/166701 2025-11-03T16:15:22.2769860Z * [new tag] ciflow/inductor/166720 -> ciflow/inductor/166720 2025-11-03T16:15:22.2770134Z * [new tag] ciflow/inductor/166738 -> ciflow/inductor/166738 2025-11-03T16:15:22.2770653Z * [new tag] ciflow/inductor/166740 -> ciflow/inductor/166740 2025-11-03T16:15:22.2770891Z * [new tag] ciflow/inductor/166741 -> ciflow/inductor/166741 2025-11-03T16:15:22.2771354Z * [new tag] ciflow/inductor/166745 -> ciflow/inductor/166745 2025-11-03T16:15:22.2772062Z * [new tag] ciflow/inductor/166747 -> ciflow/inductor/166747 2025-11-03T16:15:22.2772194Z * [new tag] ciflow/inductor/166757 -> ciflow/inductor/166757 2025-11-03T16:15:22.2773432Z * [new tag] ciflow/inductor/166762 -> ciflow/inductor/166762 2025-11-03T16:15:22.2773686Z * [new tag] ciflow/inductor/166775 -> ciflow/inductor/166775 2025-11-03T16:15:22.2773824Z * [new tag] ciflow/inductor/166776 -> ciflow/inductor/166776 2025-11-03T16:15:22.2774202Z * [new tag] ciflow/inductor/166777 -> ciflow/inductor/166777 2025-11-03T16:15:22.2776306Z * [new tag] ciflow/inductor/166788 -> ciflow/inductor/166788 2025-11-03T16:15:22.2776624Z * [new tag] ciflow/inductor/166791 -> ciflow/inductor/166791 2025-11-03T16:15:22.2776827Z * [new tag] ciflow/inductor/166793 -> ciflow/inductor/166793 2025-11-03T16:15:22.2777030Z * [new tag] ciflow/inductor/166803 -> ciflow/inductor/166803 2025-11-03T16:15:22.2777167Z * [new tag] ciflow/inductor/166808 -> ciflow/inductor/166808 2025-11-03T16:15:22.2777518Z * [new tag] ciflow/inductor/166822 -> ciflow/inductor/166822 2025-11-03T16:15:22.2778108Z * [new tag] ciflow/inductor/166834 -> ciflow/inductor/166834 2025-11-03T16:15:22.2778375Z * [new tag] ciflow/inductor/166839 -> ciflow/inductor/166839 2025-11-03T16:15:22.2778692Z * [new tag] ciflow/inductor/166846 -> ciflow/inductor/166846 2025-11-03T16:15:22.2779222Z * [new tag] ciflow/inductor/166855 -> ciflow/inductor/166855 2025-11-03T16:15:22.2779871Z * [new tag] ciflow/inductor/166867 -> ciflow/inductor/166867 2025-11-03T16:15:22.2780324Z * [new tag] ciflow/inductor/166868 -> ciflow/inductor/166868 2025-11-03T16:15:22.2780710Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-11-03T16:15:22.2781661Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-11-03T16:15:22.2781929Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-11-03T16:15:22.2782447Z * [new tag] ciflow/linux-aarch64/166380 -> ciflow/linux-aarch64/166380 2025-11-03T16:15:22.2782868Z * [new tag] ciflow/linux-aarch64/166441 -> ciflow/linux-aarch64/166441 2025-11-03T16:15:22.2783317Z * [new tag] ciflow/linux-aarch64/166549 -> ciflow/linux-aarch64/166549 2025-11-03T16:15:22.2783781Z * [new tag] ciflow/linux-aarch64/166640 -> ciflow/linux-aarch64/166640 2025-11-03T16:15:22.2784423Z * [new tag] ciflow/linux-aarch64/166691 -> ciflow/linux-aarch64/166691 2025-11-03T16:15:22.2784766Z * [new tag] ciflow/linux-aarch64/166849 -> ciflow/linux-aarch64/166849 2025-11-03T16:15:22.2788184Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-11-03T16:15:22.2788488Z * [new tag] ciflow/mps/157554 -> ciflow/mps/157554 2025-11-03T16:15:22.2788633Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-11-03T16:15:22.2788753Z * [new tag] ciflow/mps/165952 -> ciflow/mps/165952 2025-11-03T16:15:22.2789005Z * [new tag] ciflow/mps/166254 -> ciflow/mps/166254 2025-11-03T16:15:22.2789781Z * [new tag] ciflow/mps/166273 -> ciflow/mps/166273 2025-11-03T16:15:22.2789921Z * [new tag] ciflow/mps/166396 -> ciflow/mps/166396 2025-11-03T16:15:22.2790038Z * [new tag] ciflow/mps/166615 -> ciflow/mps/166615 2025-11-03T16:15:22.2790140Z * [new tag] ciflow/mps/166687 -> ciflow/mps/166687 2025-11-03T16:15:22.2790252Z * [new tag] ciflow/mps/166712 -> ciflow/mps/166712 2025-11-03T16:15:22.2790354Z * [new tag] ciflow/mps/166818 -> ciflow/mps/166818 2025-11-03T16:15:22.2790648Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-11-03T16:15:22.2790812Z * [new tag] ciflow/nightly/166727 -> ciflow/nightly/166727 2025-11-03T16:15:22.2791068Z * [new tag] ciflow/op-benchmark/165915 -> ciflow/op-benchmark/165915 2025-11-03T16:15:22.2791479Z * [new tag] ciflow/op-benchmark/166331 -> ciflow/op-benchmark/166331 2025-11-03T16:15:22.2792151Z * [new tag] ciflow/op-benchmark/166640 -> ciflow/op-benchmark/166640 2025-11-03T16:15:22.2792347Z * [new tag] ciflow/op-benchmark/166652 -> ciflow/op-benchmark/166652 2025-11-03T16:15:22.2792743Z * [new tag] ciflow/op-benchmark/166731 -> ciflow/op-benchmark/166731 2025-11-03T16:15:22.2793485Z * [new tag] ciflow/periodic-rocm-mi200/166743 -> ciflow/periodic-rocm-mi200/166743 2025-11-03T16:15:22.2794006Z * [new tag] ciflow/periodic-rocm-mi300/166517 -> ciflow/periodic-rocm-mi300/166517 2025-11-03T16:15:22.2794729Z * [new tag] ciflow/periodic-rocm-mi300/166743 -> ciflow/periodic-rocm-mi300/166743 2025-11-03T16:15:22.2794965Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-11-03T16:15:22.2798846Z * [new tag] ciflow/periodic/06e71c85583d9857f716b441eef9db40d3f27a39 -> ciflow/periodic/06e71c85583d9857f716b441eef9db40d3f27a39 2025-11-03T16:15:22.2799190Z * [new tag] ciflow/periodic/0e46a10aa7cf69354a0a0cbd268506b53a7cc882 -> ciflow/periodic/0e46a10aa7cf69354a0a0cbd268506b53a7cc882 2025-11-03T16:15:22.2799330Z * [new tag] ciflow/periodic/163490 -> ciflow/periodic/163490 2025-11-03T16:15:22.2799462Z * [new tag] ciflow/periodic/164202 -> ciflow/periodic/164202 2025-11-03T16:15:22.2799581Z * [new tag] ciflow/periodic/164938 -> ciflow/periodic/164938 2025-11-03T16:15:22.2799702Z * [new tag] ciflow/periodic/165885 -> ciflow/periodic/165885 2025-11-03T16:15:22.2799822Z * [new tag] ciflow/periodic/166517 -> ciflow/periodic/166517 2025-11-03T16:15:22.2800628Z * [new tag] ciflow/periodic/1e836bc769f9cfabc7659a172b8c2edee7c375d3 -> ciflow/periodic/1e836bc769f9cfabc7659a172b8c2edee7c375d3 2025-11-03T16:15:22.2800804Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-11-03T16:15:22.2800932Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-11-03T16:15:22.2801055Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-11-03T16:15:22.2801586Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-11-03T16:15:22.2802302Z * [new tag] ciflow/periodic/687c15c0b3f01118536413d21efcd052838fa10d -> ciflow/periodic/687c15c0b3f01118536413d21efcd052838fa10d 2025-11-03T16:15:22.2802596Z * [new tag] ciflow/periodic/7379972cc0e8a4b4d88b4bea5c8be0aeffdb076a -> ciflow/periodic/7379972cc0e8a4b4d88b4bea5c8be0aeffdb076a 2025-11-03T16:15:22.2803109Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-11-03T16:15:22.2806234Z * [new tag] ciflow/periodic/96b61844a722d2ae8b1a2dc283a6c0060b40782e -> ciflow/periodic/96b61844a722d2ae8b1a2dc283a6c0060b40782e 2025-11-03T16:15:22.2806717Z * [new tag] ciflow/periodic/be283297100ab86123e74b7a8372995d32b140c8 -> ciflow/periodic/be283297100ab86123e74b7a8372995d32b140c8 2025-11-03T16:15:22.2806999Z * [new tag] ciflow/periodic/bfc2050db975e589795cd3eceaed2e83bf89ad35 -> ciflow/periodic/bfc2050db975e589795cd3eceaed2e83bf89ad35 2025-11-03T16:15:22.2807264Z * [new tag] ciflow/periodic/c5701d0ab5f55b23243dd437f57e856970c45bb6 -> ciflow/periodic/c5701d0ab5f55b23243dd437f57e856970c45bb6 2025-11-03T16:15:22.2807541Z * [new tag] ciflow/periodic/cde81e92b95eee9af2879c9c75f7b03699ca72ad -> ciflow/periodic/cde81e92b95eee9af2879c9c75f7b03699ca72ad 2025-11-03T16:15:22.2808172Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-11-03T16:15:22.2808356Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-11-03T16:15:22.2808496Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-11-03T16:15:22.2809062Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-11-03T16:15:22.2809384Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-11-03T16:15:22.2809768Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-11-03T16:15:22.2810244Z * [new tag] ciflow/pull/166321 -> ciflow/pull/166321 2025-11-03T16:15:22.2810661Z * [new tag] ciflow/pull/166768 -> ciflow/pull/166768 2025-11-03T16:15:22.2812165Z * [new tag] ciflow/pull/18f425962656f009e151944d12f76e2f856b85df -> ciflow/pull/18f425962656f009e151944d12f76e2f856b85df 2025-11-03T16:15:22.2812744Z * [new tag] ciflow/pull/76780b1a3d208b142f067bdd655e7d4c5615efe1 -> ciflow/pull/76780b1a3d208b142f067bdd655e7d4c5615efe1 2025-11-03T16:15:22.2813128Z * [new tag] ciflow/pull/79aee77381b21d41c77148e5ff84c4b351aaf144 -> ciflow/pull/79aee77381b21d41c77148e5ff84c4b351aaf144 2025-11-03T16:15:22.2813694Z * [new tag] ciflow/pull/bc03d7c97462abfd45fc1b7b73d5c7acbbd2c185 -> ciflow/pull/bc03d7c97462abfd45fc1b7b73d5c7acbbd2c185 2025-11-03T16:15:22.2814102Z * [new tag] ciflow/pull/cee03634dab2dbf259d84bc389bdea80de0f80ac -> ciflow/pull/cee03634dab2dbf259d84bc389bdea80de0f80ac 2025-11-03T16:15:22.2814717Z * [new tag] ciflow/pull/d962bed15798961ffd10e64068f86a2cec411ffb -> ciflow/pull/d962bed15798961ffd10e64068f86a2cec411ffb 2025-11-03T16:15:22.2815366Z * [new tag] ciflow/pull/e471800dceb8a6592562fc4fc96a5e80bc494c0a -> ciflow/pull/e471800dceb8a6592562fc4fc96a5e80bc494c0a 2025-11-03T16:15:22.2815684Z * [new tag] ciflow/pull/f013e804c876a34cbc86a6a475597e3e29812883 -> ciflow/pull/f013e804c876a34cbc86a6a475597e3e29812883 2025-11-03T16:15:22.2816185Z * [new tag] ciflow/pull/f5cb9a4c68d9271c58ef4d3257210984b8e85099 -> ciflow/pull/f5cb9a4c68d9271c58ef4d3257210984b8e85099 2025-11-03T16:15:22.2816623Z * [new tag] ciflow/rocm-mi300/162053 -> ciflow/rocm-mi300/162053 2025-11-03T16:15:22.2817044Z * [new tag] ciflow/rocm-mi300/165548 -> ciflow/rocm-mi300/165548 2025-11-03T16:15:22.2818708Z * [new tag] ciflow/rocm-mi300/166317 -> ciflow/rocm-mi300/166317 2025-11-03T16:15:22.2819007Z * [new tag] ciflow/rocm-mi300/166743 -> ciflow/rocm-mi300/166743 2025-11-03T16:15:22.2819144Z * [new tag] ciflow/rocm-mi355/162053 -> ciflow/rocm-mi355/162053 2025-11-03T16:15:22.2819340Z * [new tag] ciflow/rocm-mi355/166743 -> ciflow/rocm-mi355/166743 2025-11-03T16:15:22.2819543Z * [new tag] ciflow/rocm/115316 -> ciflow/rocm/115316 2025-11-03T16:15:22.2819997Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-11-03T16:15:22.2820392Z * [new tag] ciflow/rocm/151845 -> ciflow/rocm/151845 2025-11-03T16:15:22.2820898Z * [new tag] ciflow/rocm/157553 -> ciflow/rocm/157553 2025-11-03T16:15:22.2821174Z * [new tag] ciflow/rocm/157635 -> ciflow/rocm/157635 2025-11-03T16:15:22.2821585Z * [new tag] ciflow/rocm/160685 -> ciflow/rocm/160685 2025-11-03T16:15:22.2821999Z * [new tag] ciflow/rocm/161404 -> ciflow/rocm/161404 2025-11-03T16:15:22.2822588Z * [new tag] ciflow/rocm/162053 -> ciflow/rocm/162053 2025-11-03T16:15:22.2822724Z * [new tag] ciflow/rocm/163369 -> ciflow/rocm/163369 2025-11-03T16:15:22.2823479Z * [new tag] ciflow/rocm/164657 -> ciflow/rocm/164657 2025-11-03T16:15:22.2823606Z * [new tag] ciflow/rocm/165132 -> ciflow/rocm/165132 2025-11-03T16:15:22.2826706Z * [new tag] ciflow/rocm/165541 -> ciflow/rocm/165541 2025-11-03T16:15:22.2827004Z * [new tag] ciflow/rocm/165548 -> ciflow/rocm/165548 2025-11-03T16:15:22.2827131Z * [new tag] ciflow/rocm/165952 -> ciflow/rocm/165952 2025-11-03T16:15:22.2827244Z * [new tag] ciflow/rocm/165997 -> ciflow/rocm/165997 2025-11-03T16:15:22.2827350Z * [new tag] ciflow/rocm/166250 -> ciflow/rocm/166250 2025-11-03T16:15:22.2827581Z * [new tag] ciflow/rocm/166317 -> ciflow/rocm/166317 2025-11-03T16:15:22.2827703Z * [new tag] ciflow/rocm/166377 -> ciflow/rocm/166377 2025-11-03T16:15:22.2827884Z * [new tag] ciflow/rocm/166391 -> ciflow/rocm/166391 2025-11-03T16:15:22.2828650Z * [new tag] ciflow/rocm/166515 -> ciflow/rocm/166515 2025-11-03T16:15:22.2828808Z * [new tag] ciflow/rocm/166517 -> ciflow/rocm/166517 2025-11-03T16:15:22.2828938Z * [new tag] ciflow/rocm/166526 -> ciflow/rocm/166526 2025-11-03T16:15:22.2829166Z * [new tag] ciflow/rocm/166743 -> ciflow/rocm/166743 2025-11-03T16:15:22.2829599Z * [new tag] ciflow/rocm/166744 -> ciflow/rocm/166744 2025-11-03T16:15:22.2830201Z * [new tag] ciflow/rocm/166752 -> ciflow/rocm/166752 2025-11-03T16:15:22.2830451Z * [new tag] ciflow/rocm/166764 -> ciflow/rocm/166764 2025-11-03T16:15:22.2830874Z * [new tag] ciflow/rocm/166834 -> ciflow/rocm/166834 2025-11-03T16:15:22.2833387Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-11-03T16:15:22.2833552Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-11-03T16:15:22.2833875Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-11-03T16:15:22.2834118Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-11-03T16:15:22.2834242Z * [new tag] ciflow/slow/166844 -> ciflow/slow/166844 2025-11-03T16:15:22.2834518Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-11-03T16:15:22.2835099Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-11-03T16:15:22.2835696Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-11-03T16:15:22.2836018Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-11-03T16:15:22.2837811Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-11-03T16:15:22.2838106Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-11-03T16:15:22.2838239Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-11-03T16:15:22.2838422Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-11-03T16:15:22.2839156Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-11-03T16:15:22.2839752Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-11-03T16:15:22.2840311Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-11-03T16:15:22.2840501Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-11-03T16:15:22.2843429Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-11-03T16:15:22.2843722Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-11-03T16:15:22.2844051Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-11-03T16:15:22.2844317Z * [new tag] ciflow/trunk/04d6a6f3392a87af8baf620dc0a323ffbb188c12 -> ciflow/trunk/04d6a6f3392a87af8baf620dc0a323ffbb188c12 2025-11-03T16:15:22.2844711Z * [new tag] ciflow/trunk/0573747b6af273c6ac8cf9f1d81dc56066a57445 -> ciflow/trunk/0573747b6af273c6ac8cf9f1d81dc56066a57445 2025-11-03T16:15:22.2844937Z * [new tag] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-11-03T16:15:22.2845068Z * [new tag] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-11-03T16:15:22.2845623Z * [new tag] ciflow/trunk/148328 -> ciflow/trunk/148328 2025-11-03T16:15:22.2845730Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-11-03T16:15:22.2845955Z * [new tag] ciflow/trunk/151845 -> ciflow/trunk/151845 2025-11-03T16:15:22.2846290Z * [new tag] ciflow/trunk/152624 -> ciflow/trunk/152624 2025-11-03T16:15:22.2846566Z * [new tag] ciflow/trunk/155731 -> ciflow/trunk/155731 2025-11-03T16:15:22.2847015Z * [new tag] ciflow/trunk/156812 -> ciflow/trunk/156812 2025-11-03T16:15:22.2847410Z * [new tag] ciflow/trunk/157432 -> ciflow/trunk/157432 2025-11-03T16:15:22.2847849Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-11-03T16:15:22.2848466Z * [new tag] ciflow/trunk/160417 -> ciflow/trunk/160417 2025-11-03T16:15:22.2848890Z * [new tag] ciflow/trunk/161035 -> ciflow/trunk/161035 2025-11-03T16:15:22.2849335Z * [new tag] ciflow/trunk/161771 -> ciflow/trunk/161771 2025-11-03T16:15:22.2849761Z * [new tag] ciflow/trunk/162053 -> ciflow/trunk/162053 2025-11-03T16:15:22.2851911Z * [new tag] ciflow/trunk/162454 -> ciflow/trunk/162454 2025-11-03T16:15:22.2852207Z * [new tag] ciflow/trunk/162564 -> ciflow/trunk/162564 2025-11-03T16:15:22.2852409Z * [new tag] ciflow/trunk/162709 -> ciflow/trunk/162709 2025-11-03T16:15:22.2852533Z * [new tag] ciflow/trunk/162795 -> ciflow/trunk/162795 2025-11-03T16:15:22.2852714Z * [new tag] ciflow/trunk/163249 -> ciflow/trunk/163249 2025-11-03T16:15:22.2852835Z * [new tag] ciflow/trunk/163490 -> ciflow/trunk/163490 2025-11-03T16:15:22.2853014Z * [new tag] ciflow/trunk/164202 -> ciflow/trunk/164202 2025-11-03T16:15:22.2853612Z * [new tag] ciflow/trunk/164938 -> ciflow/trunk/164938 2025-11-03T16:15:22.2853740Z * [new tag] ciflow/trunk/164979 -> ciflow/trunk/164979 2025-11-03T16:15:22.2854027Z * [new tag] ciflow/trunk/164980 -> ciflow/trunk/164980 2025-11-03T16:15:22.2854434Z * [new tag] ciflow/trunk/165029 -> ciflow/trunk/165029 2025-11-03T16:15:22.2855306Z * [new tag] ciflow/trunk/165049 -> ciflow/trunk/165049 2025-11-03T16:15:22.2855558Z * [new tag] ciflow/trunk/165067 -> ciflow/trunk/165067 2025-11-03T16:15:22.2855860Z * [new tag] ciflow/trunk/165132 -> ciflow/trunk/165132 2025-11-03T16:15:22.2860474Z * [new tag] ciflow/trunk/165216 -> ciflow/trunk/165216 2025-11-03T16:15:22.2860608Z * [new tag] ciflow/trunk/165274 -> ciflow/trunk/165274 2025-11-03T16:15:22.2860730Z * [new tag] ciflow/trunk/165431 -> ciflow/trunk/165431 2025-11-03T16:15:22.2860831Z * [new tag] ciflow/trunk/165548 -> ciflow/trunk/165548 2025-11-03T16:15:22.2860940Z * [new tag] ciflow/trunk/165642 -> ciflow/trunk/165642 2025-11-03T16:15:22.2861054Z * [new tag] ciflow/trunk/165646 -> ciflow/trunk/165646 2025-11-03T16:15:22.2861162Z * [new tag] ciflow/trunk/165728 -> ciflow/trunk/165728 2025-11-03T16:15:22.2861284Z * [new tag] ciflow/trunk/165824 -> ciflow/trunk/165824 2025-11-03T16:15:22.2861385Z * [new tag] ciflow/trunk/165885 -> ciflow/trunk/165885 2025-11-03T16:15:22.2862871Z * [new tag] ciflow/trunk/165922 -> ciflow/trunk/165922 2025-11-03T16:15:22.2863003Z * [new tag] ciflow/trunk/165972 -> ciflow/trunk/165972 2025-11-03T16:15:22.2863125Z * [new tag] ciflow/trunk/166010 -> ciflow/trunk/166010 2025-11-03T16:15:22.2863228Z * [new tag] ciflow/trunk/166071 -> ciflow/trunk/166071 2025-11-03T16:15:22.2863340Z * [new tag] ciflow/trunk/166160 -> ciflow/trunk/166160 2025-11-03T16:15:22.2863589Z * [new tag] ciflow/trunk/166170 -> ciflow/trunk/166170 2025-11-03T16:15:22.2863702Z * [new tag] ciflow/trunk/166198 -> ciflow/trunk/166198 2025-11-03T16:15:22.2863810Z * [new tag] ciflow/trunk/166294 -> ciflow/trunk/166294 2025-11-03T16:15:22.2863925Z * [new tag] ciflow/trunk/166299 -> ciflow/trunk/166299 2025-11-03T16:15:22.2864036Z * [new tag] ciflow/trunk/166317 -> ciflow/trunk/166317 2025-11-03T16:15:22.2868689Z * [new tag] ciflow/trunk/166321 -> ciflow/trunk/166321 2025-11-03T16:15:22.2868979Z * [new tag] ciflow/trunk/166361 -> ciflow/trunk/166361 2025-11-03T16:15:22.2869127Z * [new tag] ciflow/trunk/166367 -> ciflow/trunk/166367 2025-11-03T16:15:22.2869233Z * [new tag] ciflow/trunk/166376 -> ciflow/trunk/166376 2025-11-03T16:15:22.2869363Z * [new tag] ciflow/trunk/166379 -> ciflow/trunk/166379 2025-11-03T16:15:22.2869467Z * [new tag] ciflow/trunk/166380 -> ciflow/trunk/166380 2025-11-03T16:15:22.2869571Z * [new tag] ciflow/trunk/166402 -> ciflow/trunk/166402 2025-11-03T16:15:22.2869691Z * [new tag] ciflow/trunk/166416 -> ciflow/trunk/166416 2025-11-03T16:15:22.2869923Z * [new tag] ciflow/trunk/166431 -> ciflow/trunk/166431 2025-11-03T16:15:22.2870037Z * [new tag] ciflow/trunk/166433 -> ciflow/trunk/166433 2025-11-03T16:15:22.2870140Z * [new tag] ciflow/trunk/166459 -> ciflow/trunk/166459 2025-11-03T16:15:22.2870246Z * [new tag] ciflow/trunk/166469 -> ciflow/trunk/166469 2025-11-03T16:15:22.2872076Z * [new tag] ciflow/trunk/166493 -> ciflow/trunk/166493 2025-11-03T16:15:22.2872370Z * [new tag] ciflow/trunk/166526 -> ciflow/trunk/166526 2025-11-03T16:15:22.2872508Z * [new tag] ciflow/trunk/166536 -> ciflow/trunk/166536 2025-11-03T16:15:22.2872623Z * [new tag] ciflow/trunk/166541 -> ciflow/trunk/166541 2025-11-03T16:15:22.2872903Z * [new tag] ciflow/trunk/166549 -> ciflow/trunk/166549 2025-11-03T16:15:22.2873146Z * [new tag] ciflow/trunk/166560 -> ciflow/trunk/166560 2025-11-03T16:15:22.2873265Z * [new tag] ciflow/trunk/166608 -> ciflow/trunk/166608 2025-11-03T16:15:22.2873451Z * [new tag] ciflow/trunk/166610 -> ciflow/trunk/166610 2025-11-03T16:15:22.2873580Z * [new tag] ciflow/trunk/166613 -> ciflow/trunk/166613 2025-11-03T16:15:22.2874289Z * [new tag] ciflow/trunk/166640 -> ciflow/trunk/166640 2025-11-03T16:15:22.2879152Z * [new tag] ciflow/trunk/166687 -> ciflow/trunk/166687 2025-11-03T16:15:22.2879306Z * [new tag] ciflow/trunk/166688 -> ciflow/trunk/166688 2025-11-03T16:15:22.2879418Z * [new tag] ciflow/trunk/166689 -> ciflow/trunk/166689 2025-11-03T16:15:22.2879525Z * [new tag] ciflow/trunk/166715 -> ciflow/trunk/166715 2025-11-03T16:15:22.2879651Z * [new tag] ciflow/trunk/166727 -> ciflow/trunk/166727 2025-11-03T16:15:22.2879928Z * [new tag] ciflow/trunk/166744 -> ciflow/trunk/166744 2025-11-03T16:15:22.2880066Z * [new tag] ciflow/trunk/166751 -> ciflow/trunk/166751 2025-11-03T16:15:22.2880174Z * [new tag] ciflow/trunk/166768 -> ciflow/trunk/166768 2025-11-03T16:15:22.2880285Z * [new tag] ciflow/trunk/166793 -> ciflow/trunk/166793 2025-11-03T16:15:22.2880510Z * [new tag] ciflow/trunk/166805 -> ciflow/trunk/166805 2025-11-03T16:15:22.2880617Z * [new tag] ciflow/trunk/166806 -> ciflow/trunk/166806 2025-11-03T16:15:22.2880883Z * [new tag] ciflow/trunk/166813 -> ciflow/trunk/166813 2025-11-03T16:15:22.2881016Z * [new tag] ciflow/trunk/166830 -> ciflow/trunk/166830 2025-11-03T16:15:22.2881143Z * [new tag] ciflow/trunk/166839 -> ciflow/trunk/166839 2025-11-03T16:15:22.2881252Z * [new tag] ciflow/trunk/166840 -> ciflow/trunk/166840 2025-11-03T16:15:22.2881367Z * [new tag] ciflow/trunk/166844 -> ciflow/trunk/166844 2025-11-03T16:15:22.2881477Z * [new tag] ciflow/trunk/166847 -> ciflow/trunk/166847 2025-11-03T16:15:22.2881613Z * [new tag] ciflow/trunk/166861 -> ciflow/trunk/166861 2025-11-03T16:15:22.2881903Z * [new tag] ciflow/trunk/18f425962656f009e151944d12f76e2f856b85df -> ciflow/trunk/18f425962656f009e151944d12f76e2f856b85df 2025-11-03T16:15:22.2882169Z * [new tag] ciflow/trunk/76780b1a3d208b142f067bdd655e7d4c5615efe1 -> ciflow/trunk/76780b1a3d208b142f067bdd655e7d4c5615efe1 2025-11-03T16:15:22.2882424Z * [new tag] ciflow/trunk/a19e92d43355489c8d33325321e688a6e6182267 -> ciflow/trunk/a19e92d43355489c8d33325321e688a6e6182267 2025-11-03T16:15:22.2882687Z * [new tag] ciflow/trunk/bc03d7c97462abfd45fc1b7b73d5c7acbbd2c185 -> ciflow/trunk/bc03d7c97462abfd45fc1b7b73d5c7acbbd2c185 2025-11-03T16:15:22.2882948Z * [new tag] ciflow/trunk/c3dc0c7089f3204c36961acc227a3ff503530094 -> ciflow/trunk/c3dc0c7089f3204c36961acc227a3ff503530094 2025-11-03T16:15:22.2887377Z * [new tag] ciflow/trunk/cee03634dab2dbf259d84bc389bdea80de0f80ac -> ciflow/trunk/cee03634dab2dbf259d84bc389bdea80de0f80ac 2025-11-03T16:15:22.2887839Z * [new tag] ciflow/trunk/d962bed15798961ffd10e64068f86a2cec411ffb -> ciflow/trunk/d962bed15798961ffd10e64068f86a2cec411ffb 2025-11-03T16:15:22.2888236Z * [new tag] ciflow/trunk/e471800dceb8a6592562fc4fc96a5e80bc494c0a -> ciflow/trunk/e471800dceb8a6592562fc4fc96a5e80bc494c0a 2025-11-03T16:15:22.2888945Z * [new tag] ciflow/trunk/f013e804c876a34cbc86a6a475597e3e29812883 -> ciflow/trunk/f013e804c876a34cbc86a6a475597e3e29812883 2025-11-03T16:15:22.2889276Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-11-03T16:15:22.2889393Z * [new tag] ciflow/vllm/165274 -> ciflow/vllm/165274 2025-11-03T16:15:22.2889506Z * [new tag] ciflow/vllm/166494 -> ciflow/vllm/166494 2025-11-03T16:15:22.2889633Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-11-03T16:15:22.2889750Z * [new tag] ciflow/xpu/156812 -> ciflow/xpu/156812 2025-11-03T16:15:22.2889854Z * [new tag] ciflow/xpu/161246 -> ciflow/xpu/161246 2025-11-03T16:15:22.2889956Z * [new tag] ciflow/xpu/162454 -> ciflow/xpu/162454 2025-11-03T16:15:22.2890070Z * [new tag] ciflow/xpu/162475 -> ciflow/xpu/162475 2025-11-03T16:15:22.2890170Z * [new tag] ciflow/xpu/162564 -> ciflow/xpu/162564 2025-11-03T16:15:22.2890277Z * [new tag] ciflow/xpu/165049 -> ciflow/xpu/165049 2025-11-03T16:15:22.2890377Z * [new tag] ciflow/xpu/165423 -> ciflow/xpu/165423 2025-11-03T16:15:22.2890477Z * [new tag] ciflow/xpu/166292 -> ciflow/xpu/166292 2025-11-03T16:15:22.2890585Z * [new tag] ciflow/xpu/166299 -> ciflow/xpu/166299 2025-11-03T16:15:22.2890684Z * [new tag] ciflow/xpu/166376 -> ciflow/xpu/166376 2025-11-03T16:15:22.2890789Z * [new tag] ciflow/xpu/166396 -> ciflow/xpu/166396 2025-11-03T16:15:22.2895766Z * [new tag] ciflow/xpu/166424 -> ciflow/xpu/166424 2025-11-03T16:15:22.2896053Z * [new tag] ciflow/xpu/166436 -> ciflow/xpu/166436 2025-11-03T16:15:22.2896181Z * [new tag] ciflow/xpu/166495 -> ciflow/xpu/166495 2025-11-03T16:15:22.2896280Z * [new tag] ciflow/xpu/166504 -> ciflow/xpu/166504 2025-11-03T16:15:22.2896395Z * [new tag] ciflow/xpu/166613 -> ciflow/xpu/166613 2025-11-03T16:15:22.2896497Z * [new tag] ciflow/xpu/166684 -> ciflow/xpu/166684 2025-11-03T16:15:22.2896607Z * [new tag] ciflow/xpu/166830 -> ciflow/xpu/166830 2025-11-03T16:15:22.2896707Z * [new tag] ciflow/xpu/166834 -> ciflow/xpu/166834 2025-11-03T16:15:22.2896811Z * [new tag] ciflow/xpu/166839 -> ciflow/xpu/166839 2025-11-03T16:15:22.2896909Z * [new tag] ciflow/xpu/166840 -> ciflow/xpu/166840 2025-11-03T16:15:22.2897006Z * [new tag] ciflow/xpu/166847 -> ciflow/xpu/166847 2025-11-03T16:15:22.2897114Z * [new tag] ciflow/xpu/166861 -> ciflow/xpu/166861 2025-11-03T16:15:22.2897218Z * [new tag] cslpull75 -> cslpull75 2025-11-03T16:15:22.2897317Z * [new tag] cslpull76 -> cslpull76 2025-11-03T16:15:22.2897409Z * [new tag] cslpull77 -> cslpull77 2025-11-03T16:15:22.2897496Z * [new tag] cslpull78 -> cslpull78 2025-11-03T16:15:22.2897595Z * [new tag] cslpull79 -> cslpull79 2025-11-03T16:15:22.2902575Z * [new tag] cslpull80 -> cslpull80 2025-11-03T16:15:22.2902851Z * [new tag] cslpull81 -> cslpull81 2025-11-03T16:15:22.2902964Z * [new tag] cslpull82 -> cslpull82 2025-11-03T16:15:22.2903065Z * [new tag] cslpull83 -> cslpull83 2025-11-03T16:15:22.2903173Z * [new tag] cslpull84 -> cslpull84 2025-11-03T16:15:22.2903398Z * [new tag] cslpull85 -> cslpull85 2025-11-03T16:15:22.2903509Z * [new tag] cslpull86 -> cslpull86 2025-11-03T16:15:22.2903878Z * [new tag] cslpull87 -> cslpull87 2025-11-03T16:15:22.2904475Z * [new tag] cslpull88 -> cslpull88 2025-11-03T16:15:22.2904605Z * [new tag] cslpull89 -> cslpull89 2025-11-03T16:15:22.2904697Z * [new tag] cslpull90 -> cslpull90 2025-11-03T16:15:22.2904795Z * [new tag] cslpull91 -> cslpull91 2025-11-03T16:15:22.2904886Z * [new tag] cslpull92 -> cslpull92 2025-11-03T16:15:22.2904993Z * [new tag] flight_5 -> flight_5 2025-11-03T16:15:22.2905091Z * [new tag] flight_5.1 -> flight_5.1 2025-11-03T16:15:22.2905196Z * [new tag] flight_5.2 -> flight_5.2 2025-11-03T16:15:22.2905294Z * [new tag] flight_5.3 -> flight_5.3 2025-11-03T16:15:22.2910126Z * [new tag] forpull1 -> forpull1 2025-11-03T16:15:22.2910437Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-11-03T16:15:22.2910626Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-11-03T16:15:22.2910756Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-11-03T16:15:22.2910947Z * [new tag] nightly-binary -> nightly-binary 2025-11-03T16:15:22.2911088Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-11-03T16:15:22.2911273Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-11-03T16:15:22.2911677Z * [new tag] trunk/000f49551bfda0683d395e1fe67f53cfb785868f -> trunk/000f49551bfda0683d395e1fe67f53cfb785868f 2025-11-03T16:15:22.2912243Z * [new tag] trunk/009ea77234788b29910b1205679036eaae3ee3ac -> trunk/009ea77234788b29910b1205679036eaae3ee3ac 2025-11-03T16:15:22.2912459Z * [new tag] trunk/0187db88d463905e5f7dcdf7689519e2807e5a12 -> trunk/0187db88d463905e5f7dcdf7689519e2807e5a12 2025-11-03T16:15:22.2912687Z * [new tag] trunk/02095cc09db01f1fecae436ac1a6ecdf2cfca0db -> trunk/02095cc09db01f1fecae436ac1a6ecdf2cfca0db 2025-11-03T16:15:22.2912899Z * [new tag] trunk/030de07affa347a80fa9f11ab8e9234fc6d7fd4d -> trunk/030de07affa347a80fa9f11ab8e9234fc6d7fd4d 2025-11-03T16:15:22.2913115Z * [new tag] trunk/034e951b0cfb02d7b55327cd482e58cf2695dca0 -> trunk/034e951b0cfb02d7b55327cd482e58cf2695dca0 2025-11-03T16:15:22.2913474Z * [new tag] trunk/04d6a6f3392a87af8baf620dc0a323ffbb188c12 -> trunk/04d6a6f3392a87af8baf620dc0a323ffbb188c12 2025-11-03T16:15:22.2913834Z * [new tag] trunk/0573747b6af273c6ac8cf9f1d81dc56066a57445 -> trunk/0573747b6af273c6ac8cf9f1d81dc56066a57445 2025-11-03T16:15:22.2914255Z * [new tag] trunk/061fa73c97dcfcb39077ce8fc4042e7f22824b22 -> trunk/061fa73c97dcfcb39077ce8fc4042e7f22824b22 2025-11-03T16:15:22.2914574Z * [new tag] trunk/0674e0a0f14775f920296e9dfb8b61e4960bf99d -> trunk/0674e0a0f14775f920296e9dfb8b61e4960bf99d 2025-11-03T16:15:22.2919509Z * [new tag] trunk/06e71c85583d9857f716b441eef9db40d3f27a39 -> trunk/06e71c85583d9857f716b441eef9db40d3f27a39 2025-11-03T16:15:22.2921749Z * [new tag] trunk/08ae55021eafba28b304e78ef2da9a76741ce039 -> trunk/08ae55021eafba28b304e78ef2da9a76741ce039 2025-11-03T16:15:22.2921988Z * [new tag] trunk/08b0a8f11a00e0442bd3b3066617a501dca29876 -> trunk/08b0a8f11a00e0442bd3b3066617a501dca29876 2025-11-03T16:15:22.2922224Z * [new tag] trunk/08f4535378b9ab5c7ba74828bb20ad535e6de8eb -> trunk/08f4535378b9ab5c7ba74828bb20ad535e6de8eb 2025-11-03T16:15:22.2922460Z * [new tag] trunk/0918bf321c2cfaebb91c5f9e5f0ebe462962328e -> trunk/0918bf321c2cfaebb91c5f9e5f0ebe462962328e 2025-11-03T16:15:22.2922870Z * [new tag] trunk/0947765eb9208996f221dbcb088df800be3953d7 -> trunk/0947765eb9208996f221dbcb088df800be3953d7 2025-11-03T16:15:22.2923096Z * [new tag] trunk/0a3ac47c0a36d7ed619e6577294648a7c14b1ae1 -> trunk/0a3ac47c0a36d7ed619e6577294648a7c14b1ae1 2025-11-03T16:15:22.2923446Z * [new tag] trunk/0ae3e306217a67362371267bbba0ac3fa0855fdc -> trunk/0ae3e306217a67362371267bbba0ac3fa0855fdc 2025-11-03T16:15:22.2925527Z * [new tag] trunk/0d3a4f7155508656bd928527840ed155e8b1a88b -> trunk/0d3a4f7155508656bd928527840ed155e8b1a88b 2025-11-03T16:15:22.2925758Z * [new tag] trunk/0d4992c1702d9d14bc73c29d44029adb7e698eba -> trunk/0d4992c1702d9d14bc73c29d44029adb7e698eba 2025-11-03T16:15:22.2925998Z * [new tag] trunk/0d50e5d8d4132cb09508b3e35701e7b06433420e -> trunk/0d50e5d8d4132cb09508b3e35701e7b06433420e 2025-11-03T16:15:22.2926215Z * [new tag] trunk/0d81bb7f9cef1b735ff373a245aaa2bc037b31b0 -> trunk/0d81bb7f9cef1b735ff373a245aaa2bc037b31b0 2025-11-03T16:15:22.2926431Z * [new tag] trunk/0e19561e235372a3fbed81338fa4e674116c0448 -> trunk/0e19561e235372a3fbed81338fa4e674116c0448 2025-11-03T16:15:22.2926642Z * [new tag] trunk/0e46a10aa7cf69354a0a0cbd268506b53a7cc882 -> trunk/0e46a10aa7cf69354a0a0cbd268506b53a7cc882 2025-11-03T16:15:22.2926862Z * [new tag] trunk/0eacd934bc1c1bee002fcd084300d596d789d166 -> trunk/0eacd934bc1c1bee002fcd084300d596d789d166 2025-11-03T16:15:22.2927065Z * [new tag] trunk/0ec054982304473b51e6ded40c2dc3993d4036be -> trunk/0ec054982304473b51e6ded40c2dc3993d4036be 2025-11-03T16:15:22.2927263Z * [new tag] trunk/108bb224f77842593009214ebf6258030b934642 -> trunk/108bb224f77842593009214ebf6258030b934642 2025-11-03T16:15:22.2927662Z * [new tag] trunk/110efe4df4759006ce5aad4ae1b87d93dcf33c3b -> trunk/110efe4df4759006ce5aad4ae1b87d93dcf33c3b 2025-11-03T16:15:22.2928063Z * [new tag] trunk/112960541529ab0d27bb8aa8b487ef194e048494 -> trunk/112960541529ab0d27bb8aa8b487ef194e048494 2025-11-03T16:15:22.2928291Z * [new tag] trunk/121235956bab7430fb8d080cee209607f8387ead -> trunk/121235956bab7430fb8d080cee209607f8387ead 2025-11-03T16:15:22.2928499Z * [new tag] trunk/12577064dddfc6f5daf66c5b5a73cb418a588f20 -> trunk/12577064dddfc6f5daf66c5b5a73cb418a588f20 2025-11-03T16:15:22.2928711Z * [new tag] trunk/12c12466b082834d23c1f512130da9280badfd28 -> trunk/12c12466b082834d23c1f512130da9280badfd28 2025-11-03T16:15:22.2928918Z * [new tag] trunk/13413b3b07cc72fa9c2671b2535f7e54c1b30ca2 -> trunk/13413b3b07cc72fa9c2671b2535f7e54c1b30ca2 2025-11-03T16:15:22.2929131Z * [new tag] trunk/13549e0e105291721f6f80ceb8f520ca272b971d -> trunk/13549e0e105291721f6f80ceb8f520ca272b971d 2025-11-03T16:15:22.2929340Z * [new tag] trunk/14102fb1f337d01ca61584011f5f657d14da5de6 -> trunk/14102fb1f337d01ca61584011f5f657d14da5de6 2025-11-03T16:15:22.2929563Z * [new tag] trunk/1425b40f29ab11ce21f7a98de9e35551213efe27 -> trunk/1425b40f29ab11ce21f7a98de9e35551213efe27 2025-11-03T16:15:22.2929778Z * [new tag] trunk/14d4a77495dc80da9cd7c5c29b21aaf4613ddc5b -> trunk/14d4a77495dc80da9cd7c5c29b21aaf4613ddc5b 2025-11-03T16:15:22.2929987Z * [new tag] trunk/160ab53dd57e67b3574763615cf8b33249e9afa5 -> trunk/160ab53dd57e67b3574763615cf8b33249e9afa5 2025-11-03T16:15:22.2930212Z * [new tag] trunk/16212f0d6bc1c4fb6ac0498badbe2fb74d5ff47b -> trunk/16212f0d6bc1c4fb6ac0498badbe2fb74d5ff47b 2025-11-03T16:15:22.2930424Z * [new tag] trunk/1656b253c5fcc34a53a9991303170892c6d90c1d -> trunk/1656b253c5fcc34a53a9991303170892c6d90c1d 2025-11-03T16:15:22.2930642Z * [new tag] trunk/173bcda436e9a00aa8215a414efc0c06cf4840a8 -> trunk/173bcda436e9a00aa8215a414efc0c06cf4840a8 2025-11-03T16:15:22.2930842Z * [new tag] trunk/1764f3a9c8eea4cf0aba853348446948597b7b78 -> trunk/1764f3a9c8eea4cf0aba853348446948597b7b78 2025-11-03T16:15:22.2931102Z * [new tag] trunk/17bdb232e1b3df3435fc6f54d3d713404ba96726 -> trunk/17bdb232e1b3df3435fc6f54d3d713404ba96726 2025-11-03T16:15:22.2931300Z * [new tag] trunk/17d5aa47674c5195a3d4919169fb6a334598e2a6 -> trunk/17d5aa47674c5195a3d4919169fb6a334598e2a6 2025-11-03T16:15:22.2931511Z * [new tag] trunk/181ee3bd42447b71a1a8435bf16c0877c4bc3ae7 -> trunk/181ee3bd42447b71a1a8435bf16c0877c4bc3ae7 2025-11-03T16:15:22.2935170Z * [new tag] trunk/18f425962656f009e151944d12f76e2f856b85df -> trunk/18f425962656f009e151944d12f76e2f856b85df 2025-11-03T16:15:22.2935592Z * [new tag] trunk/1abfa5f70b9aa0dcbd83dc40b092dda1cdcc7805 -> trunk/1abfa5f70b9aa0dcbd83dc40b092dda1cdcc7805 2025-11-03T16:15:22.2935954Z * [new tag] trunk/1aef88c72d3aef629b20e97a188c9dc4bab46a1a -> trunk/1aef88c72d3aef629b20e97a188c9dc4bab46a1a 2025-11-03T16:15:22.2936311Z * [new tag] trunk/1b655a87ef137d2cc9603a982532c5e033432daa -> trunk/1b655a87ef137d2cc9603a982532c5e033432daa 2025-11-03T16:15:22.2937046Z * [new tag] trunk/1c4ced2eafa470832f2f4e24487785c5b7b188c9 -> trunk/1c4ced2eafa470832f2f4e24487785c5b7b188c9 2025-11-03T16:15:22.2937291Z * [new tag] trunk/1dd6b76914397d466a7d9bfff551b6d8299fdce4 -> trunk/1dd6b76914397d466a7d9bfff551b6d8299fdce4 2025-11-03T16:15:22.2937513Z * [new tag] trunk/1dec8a67a8dc88736e9c476a3eaee301684094cd -> trunk/1dec8a67a8dc88736e9c476a3eaee301684094cd 2025-11-03T16:15:22.2937714Z * [new tag] trunk/1e3600b5287346b29a835ad67f8b33945e0ec698 -> trunk/1e3600b5287346b29a835ad67f8b33945e0ec698 2025-11-03T16:15:22.2938106Z * [new tag] trunk/1e836bc769f9cfabc7659a172b8c2edee7c375d3 -> trunk/1e836bc769f9cfabc7659a172b8c2edee7c375d3 2025-11-03T16:15:22.2938333Z * [new tag] trunk/1fa520ea654f5fc0b3c65ce6e056dd73442dd65d -> trunk/1fa520ea654f5fc0b3c65ce6e056dd73442dd65d 2025-11-03T16:15:22.2938574Z * [new tag] trunk/1fdef664a5dffc3bd3d9eba9d78ed458a5f9dbcb -> trunk/1fdef664a5dffc3bd3d9eba9d78ed458a5f9dbcb 2025-11-03T16:15:22.2938784Z * [new tag] trunk/20be077085daf739af76325d04f649dcd908316c -> trunk/20be077085daf739af76325d04f649dcd908316c 2025-11-03T16:15:22.2938996Z * [new tag] trunk/21b48f8dfa7685699df4c97c0ba373d5364230d9 -> trunk/21b48f8dfa7685699df4c97c0ba373d5364230d9 2025-11-03T16:15:22.2939213Z * [new tag] trunk/23669d02a67023ef3192f7dca0ac0547436cae50 -> trunk/23669d02a67023ef3192f7dca0ac0547436cae50 2025-11-03T16:15:22.2939423Z * [new tag] trunk/236ce736a1a84352b21b564e3e5b0be833d7c10c -> trunk/236ce736a1a84352b21b564e3e5b0be833d7c10c 2025-11-03T16:15:22.2939648Z * [new tag] trunk/239e7b541a94d16a33a4a48d3edc9a561777d86a -> trunk/239e7b541a94d16a33a4a48d3edc9a561777d86a 2025-11-03T16:15:22.2939852Z * [new tag] trunk/23b57a445c6394d46a6b46e32b7787663f40d13b -> trunk/23b57a445c6394d46a6b46e32b7787663f40d13b 2025-11-03T16:15:22.2943958Z * [new tag] trunk/24b6eb7727f7eef078f9366302f2aa3e1fef1db4 -> trunk/24b6eb7727f7eef078f9366302f2aa3e1fef1db4 2025-11-03T16:15:22.2944364Z * [new tag] trunk/24e94e021a5a96bf16397ea66243070a7be4aaef -> trunk/24e94e021a5a96bf16397ea66243070a7be4aaef 2025-11-03T16:15:22.2944722Z * [new tag] trunk/26534e9809eb2f7cd804fde5152cdd13dda2293f -> trunk/26534e9809eb2f7cd804fde5152cdd13dda2293f 2025-11-03T16:15:22.2945071Z * [new tag] trunk/267d0197bfca0232488d51dd1ff735d619adc2cf -> trunk/267d0197bfca0232488d51dd1ff735d619adc2cf 2025-11-03T16:15:22.2945719Z * [new tag] trunk/2699f5410b6802277db724f145c281814d6c9ac4 -> trunk/2699f5410b6802277db724f145c281814d6c9ac4 2025-11-03T16:15:22.2945976Z * [new tag] trunk/2829d48bd113043a98e81b4ab7fa0f3ca9ae0072 -> trunk/2829d48bd113043a98e81b4ab7fa0f3ca9ae0072 2025-11-03T16:15:22.2946360Z * [new tag] trunk/284716a691580cf0508a7c5a4f9f7306a32092ad -> trunk/284716a691580cf0508a7c5a4f9f7306a32092ad 2025-11-03T16:15:22.2946570Z * [new tag] trunk/298666631baa6634666ac4dd4599f1d52bafa718 -> trunk/298666631baa6634666ac4dd4599f1d52bafa718 2025-11-03T16:15:22.2946791Z * [new tag] trunk/2a058bfecf4f4a47927b4eaae9324c4999a8a794 -> trunk/2a058bfecf4f4a47927b4eaae9324c4999a8a794 2025-11-03T16:15:22.2947022Z * [new tag] trunk/2a5f87decf34b3d0ea7670238e2fd4620ed19e9f -> trunk/2a5f87decf34b3d0ea7670238e2fd4620ed19e9f 2025-11-03T16:15:22.2947243Z * [new tag] trunk/2c9f877fa71c90a1313deeda4b4486d37e72a8a9 -> trunk/2c9f877fa71c90a1313deeda4b4486d37e72a8a9 2025-11-03T16:15:22.2947468Z * [new tag] trunk/2ce894bb1d3e9698306b87091c900a72acac75ea -> trunk/2ce894bb1d3e9698306b87091c900a72acac75ea 2025-11-03T16:15:22.2947678Z * [new tag] trunk/2dc56456cbb2645c37c3176b7d9a263933bdec27 -> trunk/2dc56456cbb2645c37c3176b7d9a263933bdec27 2025-11-03T16:15:22.2947907Z * [new tag] trunk/2de4cf2102fdc35da00cc8070fdbcaf57198a576 -> trunk/2de4cf2102fdc35da00cc8070fdbcaf57198a576 2025-11-03T16:15:22.2948117Z * [new tag] trunk/2df2c316e2929519a5bad029936e9e37ed637b12 -> trunk/2df2c316e2929519a5bad029936e9e37ed637b12 2025-11-03T16:15:22.2948340Z * [new tag] trunk/30157d30f0e0db6a4fa7a2a3ea75e515645e36de -> trunk/30157d30f0e0db6a4fa7a2a3ea75e515645e36de 2025-11-03T16:15:22.2948593Z * [new tag] trunk/3041ede082d3b0f33bec0cef5062eabc5173b446 -> trunk/3041ede082d3b0f33bec0cef5062eabc5173b446 2025-11-03T16:15:22.2948857Z * [new tag] trunk/311ea0dec0c50f395e6dac7b3875e81ee243fceb -> trunk/311ea0dec0c50f395e6dac7b3875e81ee243fceb 2025-11-03T16:15:22.2949083Z * [new tag] trunk/31e42eb732d32e71c9954d6b2104cf1d129370d5 -> trunk/31e42eb732d32e71c9954d6b2104cf1d129370d5 2025-11-03T16:15:22.2949290Z * [new tag] trunk/32066772b3dee643b1657b8957f32b5ac8b1390a -> trunk/32066772b3dee643b1657b8957f32b5ac8b1390a 2025-11-03T16:15:22.2949506Z * [new tag] trunk/32920926f07e573083ecf81a40c898f47f4df631 -> trunk/32920926f07e573083ecf81a40c898f47f4df631 2025-11-03T16:15:22.2949728Z * [new tag] trunk/32fe4f681e2bfb2fdecf20027e29e1aeb6ad5297 -> trunk/32fe4f681e2bfb2fdecf20027e29e1aeb6ad5297 2025-11-03T16:15:22.2949946Z * [new tag] trunk/335b5c7d4bf3295d517902370142f007ca024cd0 -> trunk/335b5c7d4bf3295d517902370142f007ca024cd0 2025-11-03T16:15:22.2950168Z * [new tag] trunk/34d6ef7022be252ce18a9673f75eed1732dff22a -> trunk/34d6ef7022be252ce18a9673f75eed1732dff22a 2025-11-03T16:15:22.2954306Z * [new tag] trunk/35f3572fa483a8edb101d5765564e1ae274f3d45 -> trunk/35f3572fa483a8edb101d5765564e1ae274f3d45 2025-11-03T16:15:22.2954556Z * [new tag] trunk/365ed62f61dde7fc14dca2199692555e5d1abbe1 -> trunk/365ed62f61dde7fc14dca2199692555e5d1abbe1 2025-11-03T16:15:22.2954930Z * [new tag] trunk/369f2d6951d4c12571eb03b95d74ef2f21296847 -> trunk/369f2d6951d4c12571eb03b95d74ef2f21296847 2025-11-03T16:15:22.2955264Z * [new tag] trunk/36a48e7e6d2d01cd47f002140589a1ac2580ef3f -> trunk/36a48e7e6d2d01cd47f002140589a1ac2580ef3f 2025-11-03T16:15:22.2955597Z * [new tag] trunk/3895ce093f3e71b768d032a28a949a9f390c0d00 -> trunk/3895ce093f3e71b768d032a28a949a9f390c0d00 2025-11-03T16:15:22.2956380Z * [new tag] trunk/392acee68a5ca90279109df023583edaef73d50f -> trunk/392acee68a5ca90279109df023583edaef73d50f 2025-11-03T16:15:22.2956624Z * [new tag] trunk/398775a43e9808205f75c81d36f5087117d3f3f4 -> trunk/398775a43e9808205f75c81d36f5087117d3f3f4 2025-11-03T16:15:22.2956869Z * [new tag] trunk/398fdd32bbf7b5f8a2b67e637c8a2e46572d9f28 -> trunk/398fdd32bbf7b5f8a2b67e637c8a2e46572d9f28 2025-11-03T16:15:22.2957087Z * [new tag] trunk/39e5cdddf7e57881c52473d1288a66f0222527e1 -> trunk/39e5cdddf7e57881c52473d1288a66f0222527e1 2025-11-03T16:15:22.2957446Z * [new tag] trunk/3b4315940d371385674f957f810cb0f1ca0060bd -> trunk/3b4315940d371385674f957f810cb0f1ca0060bd 2025-11-03T16:15:22.2957682Z * [new tag] trunk/3b5d38a3bc55c32232cee8cf87e492c965f276fd -> trunk/3b5d38a3bc55c32232cee8cf87e492c965f276fd 2025-11-03T16:15:22.2962791Z * [new tag] trunk/3ca216ae172e35adde34a319a1a01faaf218e7c5 -> trunk/3ca216ae172e35adde34a319a1a01faaf218e7c5 2025-11-03T16:15:22.2963203Z * [new tag] trunk/3cc5949dc2c6e588d42cff36121819a8539902a7 -> trunk/3cc5949dc2c6e588d42cff36121819a8539902a7 2025-11-03T16:15:22.2963578Z * [new tag] trunk/3d4a2d8a93ad57e9fd1c4e5a03dd431dd0123bb3 -> trunk/3d4a2d8a93ad57e9fd1c4e5a03dd431dd0123bb3 2025-11-03T16:15:22.2963940Z * [new tag] trunk/3d4ca228beb851e2f1cc24d8300804d5884973c1 -> trunk/3d4ca228beb851e2f1cc24d8300804d5884973c1 2025-11-03T16:15:22.2964706Z * [new tag] trunk/3dc92d69ed40fd952244e54bbda0240928756654 -> trunk/3dc92d69ed40fd952244e54bbda0240928756654 2025-11-03T16:15:22.2964974Z * [new tag] trunk/3eddf049221fc04c2ac9d4af53c00305484ef325 -> trunk/3eddf049221fc04c2ac9d4af53c00305484ef325 2025-11-03T16:15:22.2965212Z * [new tag] trunk/3f1824742cac2ffb9a3afd90953c492c6c7f2f50 -> trunk/3f1824742cac2ffb9a3afd90953c492c6c7f2f50 2025-11-03T16:15:22.2965426Z * [new tag] trunk/3f5401020b3fd66238a5eda31576b898d086e14e -> trunk/3f5401020b3fd66238a5eda31576b898d086e14e 2025-11-03T16:15:22.2965656Z * [new tag] trunk/3f6538febd727b782e6e13cfd026a309fb14351d -> trunk/3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:15:22.2966037Z * [new tag] trunk/3f69b4d9b46b1de627896c457bce1937e53d82e5 -> trunk/3f69b4d9b46b1de627896c457bce1937e53d82e5 2025-11-03T16:15:22.2966273Z * [new tag] trunk/401c2f965726fda8f78b08f3745e11008b38e7bf -> trunk/401c2f965726fda8f78b08f3745e11008b38e7bf 2025-11-03T16:15:22.2966506Z * [new tag] trunk/4295a9a1587cc62a3ef4ad9b1e552c78471d5664 -> trunk/4295a9a1587cc62a3ef4ad9b1e552c78471d5664 2025-11-03T16:15:22.2966751Z * [new tag] trunk/4316df857c9e7f301142eb54d06a85a43f8d617b -> trunk/4316df857c9e7f301142eb54d06a85a43f8d617b 2025-11-03T16:15:22.2966973Z * [new tag] trunk/43c30f607eeca0d3e9a26911d9c2131fc250eadd -> trunk/43c30f607eeca0d3e9a26911d9c2131fc250eadd 2025-11-03T16:15:22.2967191Z * [new tag] trunk/45c3f02d69b0a8447381fbc3f3f8731ab0d48ed1 -> trunk/45c3f02d69b0a8447381fbc3f3f8731ab0d48ed1 2025-11-03T16:15:22.2967422Z * [new tag] trunk/467c21ad9ae4133c20a3c098a0355e9ac20d48aa -> trunk/467c21ad9ae4133c20a3c098a0355e9ac20d48aa 2025-11-03T16:15:22.2967645Z * [new tag] trunk/46d17e887150c5de8a74314b6d9fcba1feca259b -> trunk/46d17e887150c5de8a74314b6d9fcba1feca259b 2025-11-03T16:15:22.2967870Z * [new tag] trunk/476b149a00cf52ceba2b2d8b6e14989d2bf540e2 -> trunk/476b149a00cf52ceba2b2d8b6e14989d2bf540e2 2025-11-03T16:15:22.2968104Z * [new tag] trunk/47ec1e9990b89bf9705dc463cad587bd357bc848 -> trunk/47ec1e9990b89bf9705dc463cad587bd357bc848 2025-11-03T16:15:22.2968324Z * [new tag] trunk/47f00243106918eec3ef9119272977fdf2864ad8 -> trunk/47f00243106918eec3ef9119272977fdf2864ad8 2025-11-03T16:15:22.2968548Z * [new tag] trunk/47f50cfd456313d8b46fcc7a1f6de477aa0a5aee -> trunk/47f50cfd456313d8b46fcc7a1f6de477aa0a5aee 2025-11-03T16:15:22.2968769Z * [new tag] trunk/483845a9c42267a6a139b7382e5ee340502376f8 -> trunk/483845a9c42267a6a139b7382e5ee340502376f8 2025-11-03T16:15:22.2968999Z * [new tag] trunk/48c3b71ecc245cb422ec9ab4c7a2078f7ef0b3df -> trunk/48c3b71ecc245cb422ec9ab4c7a2078f7ef0b3df 2025-11-03T16:15:22.2969219Z * [new tag] trunk/48e672d14994c912caccba0c7f2f5df54c0d86b3 -> trunk/48e672d14994c912caccba0c7f2f5df54c0d86b3 2025-11-03T16:15:22.2969518Z * [new tag] trunk/4a7bc1d522bed4f3f792b4641372ec3d06b79b5d -> trunk/4a7bc1d522bed4f3f792b4641372ec3d06b79b5d 2025-11-03T16:15:22.2969932Z * [new tag] trunk/4a7fefd7c783913b811f49e5d6242086758dca6c -> trunk/4a7fefd7c783913b811f49e5d6242086758dca6c 2025-11-03T16:15:22.2970292Z * [new tag] trunk/4a94591321a62b31658f9259680de87e4be9e50d -> trunk/4a94591321a62b31658f9259680de87e4be9e50d 2025-11-03T16:15:22.2970652Z * [new tag] trunk/4acc66f1192ab7743abcc50383aefc5447447f9d -> trunk/4acc66f1192ab7743abcc50383aefc5447447f9d 2025-11-03T16:15:22.2970999Z * [new tag] trunk/4c38887346b6a6185bb3e170621dc13d5f34e9ce -> trunk/4c38887346b6a6185bb3e170621dc13d5f34e9ce 2025-11-03T16:15:22.2971344Z * [new tag] trunk/4cc64d62346edb2e13320f20740f41ba41f64c70 -> trunk/4cc64d62346edb2e13320f20740f41ba41f64c70 2025-11-03T16:15:22.2971684Z * [new tag] trunk/4e6afa8c07b73e0b339e3baacbc73102748e61dd -> trunk/4e6afa8c07b73e0b339e3baacbc73102748e61dd 2025-11-03T16:15:22.2972031Z * [new tag] trunk/4e7232c5daf753e04e8f4189229e3c33888a33e5 -> trunk/4e7232c5daf753e04e8f4189229e3c33888a33e5 2025-11-03T16:15:22.2976563Z * [new tag] trunk/4e8ba37ce3320cac033019131b817b4aca29699b -> trunk/4e8ba37ce3320cac033019131b817b4aca29699b 2025-11-03T16:15:22.2976973Z * [new tag] trunk/4fada51ada8b3190971f2f47d97346057320866a -> trunk/4fada51ada8b3190971f2f47d97346057320866a 2025-11-03T16:15:22.2977335Z * [new tag] trunk/5016e7b2eb73589e35c7a8f0f0ea3bfba221f17d -> trunk/5016e7b2eb73589e35c7a8f0f0ea3bfba221f17d 2025-11-03T16:15:22.2977828Z * [new tag] trunk/51667435f50c025ca3655ff0eeb917b4ef0ffb78 -> trunk/51667435f50c025ca3655ff0eeb917b4ef0ffb78 2025-11-03T16:15:22.2978072Z * [new tag] trunk/52db60170dd192afa48233f03574a8d8bcd39d35 -> trunk/52db60170dd192afa48233f03574a8d8bcd39d35 2025-11-03T16:15:22.2978322Z * [new tag] trunk/544b443ea1d1a9b19e65f981168a01cb87a2d333 -> trunk/544b443ea1d1a9b19e65f981168a01cb87a2d333 2025-11-03T16:15:22.2978546Z * [new tag] trunk/551921d484d209e9a95d77262b1472bc1d18422d -> trunk/551921d484d209e9a95d77262b1472bc1d18422d 2025-11-03T16:15:22.2978776Z * [new tag] trunk/56838bad5f197d32c0ee77ce2eaa7a0cb462af68 -> trunk/56838bad5f197d32c0ee77ce2eaa7a0cb462af68 2025-11-03T16:15:22.2979001Z * [new tag] trunk/56a809aa0765ec2edea57ab37f92b1e78a214dcc -> trunk/56a809aa0765ec2edea57ab37f92b1e78a214dcc 2025-11-03T16:15:22.2979241Z * [new tag] trunk/56afad4eb300be2008d8b24581bcddcd62021036 -> trunk/56afad4eb300be2008d8b24581bcddcd62021036 2025-11-03T16:15:22.2979477Z * [new tag] trunk/572cc12b42f6edf76affd45ea9a23f4c8db5c7e1 -> trunk/572cc12b42f6edf76affd45ea9a23f4c8db5c7e1 2025-11-03T16:15:22.2979705Z * [new tag] trunk/5849eea129b47556630aac4276529afe287b91e1 -> trunk/5849eea129b47556630aac4276529afe287b91e1 2025-11-03T16:15:22.2979939Z * [new tag] trunk/59ddfb69a731970bf5762fbea43ff2987425d3ea -> trunk/59ddfb69a731970bf5762fbea43ff2987425d3ea 2025-11-03T16:15:22.2980165Z * [new tag] trunk/5a3930abbc19eac9a179455df82e206e69765ed2 -> trunk/5a3930abbc19eac9a179455df82e206e69765ed2 2025-11-03T16:15:22.2980391Z * [new tag] trunk/5bcfdae71da2f405a89bace6b09d616107cc3965 -> trunk/5bcfdae71da2f405a89bace6b09d616107cc3965 2025-11-03T16:15:22.2980630Z * [new tag] trunk/5cbdade914d6bc9f67827ef917974980a74e9372 -> trunk/5cbdade914d6bc9f67827ef917974980a74e9372 2025-11-03T16:15:22.2980855Z * [new tag] trunk/5cdbcb52334222d1a2667dabd6da993d1e9ddc93 -> trunk/5cdbcb52334222d1a2667dabd6da993d1e9ddc93 2025-11-03T16:15:22.2985494Z * [new tag] trunk/5d0b3e28dc18fac3c09edb24078998371f878f82 -> trunk/5d0b3e28dc18fac3c09edb24078998371f878f82 2025-11-03T16:15:22.2985916Z * [new tag] trunk/5d288bc3f73873887f681e15af83c5525e6a60bd -> trunk/5d288bc3f73873887f681e15af83c5525e6a60bd 2025-11-03T16:15:22.2986429Z * [new tag] trunk/5d6230779d920e9c449c22d22fd0f69fe7b73632 -> trunk/5d6230779d920e9c449c22d22fd0f69fe7b73632 2025-11-03T16:15:22.2987186Z * [new tag] trunk/5e05a0ae99c320bc570a3a42c5fcd6bbf2b2e9b4 -> trunk/5e05a0ae99c320bc570a3a42c5fcd6bbf2b2e9b4 2025-11-03T16:15:22.2987455Z * [new tag] trunk/5e7272b60a5d7203fd1d048ecac6f202526cac63 -> trunk/5e7272b60a5d7203fd1d048ecac6f202526cac63 2025-11-03T16:15:22.2987682Z * [new tag] trunk/5e769ff86780a7ffd561615dbf4b0defe80cfbb9 -> trunk/5e769ff86780a7ffd561615dbf4b0defe80cfbb9 2025-11-03T16:15:22.2987922Z * [new tag] trunk/5ebf74a655b3335fb079ee03f41228a77bcc630e -> trunk/5ebf74a655b3335fb079ee03f41228a77bcc630e 2025-11-03T16:15:22.2988132Z * [new tag] trunk/5fc2c7a2a11444c31338da9134452810f3ca8acd -> trunk/5fc2c7a2a11444c31338da9134452810f3ca8acd 2025-11-03T16:15:22.2988377Z * [new tag] trunk/5fd1d41e62ce4436c2bb08c7f4702df0021e2c62 -> trunk/5fd1d41e62ce4436c2bb08c7f4702df0021e2c62 2025-11-03T16:15:22.2988585Z * [new tag] trunk/60333de85dec845d6a0e651764c16a730a27828d -> trunk/60333de85dec845d6a0e651764c16a730a27828d 2025-11-03T16:15:22.2988791Z * [new tag] trunk/6096c0fc746dc99189510a1845fd45c667178c03 -> trunk/6096c0fc746dc99189510a1845fd45c667178c03 2025-11-03T16:15:22.2989011Z * [new tag] trunk/60bcb4ee8844e7bac7278ff662c7d5735a326758 -> trunk/60bcb4ee8844e7bac7278ff662c7d5735a326758 2025-11-03T16:15:22.2989216Z * [new tag] trunk/610c09f8f47585e811945e48abc895f88e09bda3 -> trunk/610c09f8f47585e811945e48abc895f88e09bda3 2025-11-03T16:15:22.2989571Z * [new tag] trunk/61bad3c1ea718311d7c256b05614d207d73276e8 -> trunk/61bad3c1ea718311d7c256b05614d207d73276e8 2025-11-03T16:15:22.2989792Z * [new tag] trunk/61bcc8d75ad065a40b2d7953046616d5979233b9 -> trunk/61bcc8d75ad065a40b2d7953046616d5979233b9 2025-11-03T16:15:22.2990028Z * [new tag] trunk/6268883f9c081b88b611ef01be3821503ad180ca -> trunk/6268883f9c081b88b611ef01be3821503ad180ca 2025-11-03T16:15:22.2990242Z * [new tag] trunk/629293f568229adaf60cdaf57fe69306d7782c2d -> trunk/629293f568229adaf60cdaf57fe69306d7782c2d 2025-11-03T16:15:22.2990461Z * [new tag] trunk/639a0b12395e72a8a220dc8443317f4ab3b3f1a0 -> trunk/639a0b12395e72a8a220dc8443317f4ab3b3f1a0 2025-11-03T16:15:22.2990677Z * [new tag] trunk/6530bc70fbc7d600ab0e600c02ecd9b341ce8237 -> trunk/6530bc70fbc7d600ab0e600c02ecd9b341ce8237 2025-11-03T16:15:22.2990900Z * [new tag] trunk/657f8c3e21bd8901dd8ce79ca9a54a45b27f604f -> trunk/657f8c3e21bd8901dd8ce79ca9a54a45b27f604f 2025-11-03T16:15:22.2991122Z * [new tag] trunk/65868156c613521e9635eb3f8eb3f404826ac1f9 -> trunk/65868156c613521e9635eb3f8eb3f404826ac1f9 2025-11-03T16:15:22.2991501Z * [new tag] trunk/687c15c0b3f01118536413d21efcd052838fa10d -> trunk/687c15c0b3f01118536413d21efcd052838fa10d 2025-11-03T16:15:22.2991893Z * [new tag] trunk/68b3984b77a19ed2b2afcbcae201e8b06e20f8bc -> trunk/68b3984b77a19ed2b2afcbcae201e8b06e20f8bc 2025-11-03T16:15:22.2992367Z * [new tag] trunk/694d205143b03cb26626cb86ecb399df6d5e3dd8 -> trunk/694d205143b03cb26626cb86ecb399df6d5e3dd8 2025-11-03T16:15:22.2992999Z * [new tag] trunk/694db5f54927697c9e914d35029f7e5bd9b85b96 -> trunk/694db5f54927697c9e914d35029f7e5bd9b85b96 2025-11-03T16:15:22.2993527Z * [new tag] trunk/695cb0d342f67db563fb57ce1a5f3261b073b790 -> trunk/695cb0d342f67db563fb57ce1a5f3261b073b790 2025-11-03T16:15:22.2994301Z * [new tag] trunk/69be99ee51bd0439fead4c83d1ed3eea9e1e073e -> trunk/69be99ee51bd0439fead4c83d1ed3eea9e1e073e 2025-11-03T16:15:22.2994782Z * [new tag] trunk/69fb3ebb5dbec0bd4cfd73cc9a6c870b7f604840 -> trunk/69fb3ebb5dbec0bd4cfd73cc9a6c870b7f604840 2025-11-03T16:15:22.2995371Z * [new tag] trunk/6a5a436624d08feff1de191e61eb4d4773f73bdb -> trunk/6a5a436624d08feff1de191e61eb4d4773f73bdb 2025-11-03T16:15:22.2995849Z * [new tag] trunk/6c7cad697287dff7e47545ae4b1df1e407185868 -> trunk/6c7cad697287dff7e47545ae4b1df1e407185868 2025-11-03T16:15:22.2996404Z * [new tag] trunk/6d5e651a5024acdbaf9633d9d1d9dc33c6a283c5 -> trunk/6d5e651a5024acdbaf9633d9d1d9dc33c6a283c5 2025-11-03T16:15:22.2996936Z * [new tag] trunk/6ecd6b23b6d2ab17da269ceebe9f944153f67083 -> trunk/6ecd6b23b6d2ab17da269ceebe9f944153f67083 2025-11-03T16:15:22.2997423Z * [new tag] trunk/7045aab143b6d89c2e7de11bc8cbf919d8fd78d0 -> trunk/7045aab143b6d89c2e7de11bc8cbf919d8fd78d0 2025-11-03T16:15:22.2998026Z * [new tag] trunk/70aeb491987401a380914a72c1b9d19e4c882429 -> trunk/70aeb491987401a380914a72c1b9d19e4c882429 2025-11-03T16:15:22.2998420Z * [new tag] trunk/7379972cc0e8a4b4d88b4bea5c8be0aeffdb076a -> trunk/7379972cc0e8a4b4d88b4bea5c8be0aeffdb076a 2025-11-03T16:15:22.2998966Z * [new tag] trunk/73da7a40b6dd0509b179f0ca92d9fa79973ee306 -> trunk/73da7a40b6dd0509b179f0ca92d9fa79973ee306 2025-11-03T16:15:22.2999472Z * [new tag] trunk/74336f8c77f1a906314c274e43239306c8077e90 -> trunk/74336f8c77f1a906314c274e43239306c8077e90 2025-11-03T16:15:22.3000015Z * [new tag] trunk/748162223768fb9f7daaca1f7f93d0bc37be4de9 -> trunk/748162223768fb9f7daaca1f7f93d0bc37be4de9 2025-11-03T16:15:22.3002462Z * [new tag] trunk/753d9bd8063356c0e31517ef9aabd814efd754c5 -> trunk/753d9bd8063356c0e31517ef9aabd814efd754c5 2025-11-03T16:15:22.3003068Z * [new tag] trunk/7563f61cc8a40a5ba21a498a2d98895b4eec3f39 -> trunk/7563f61cc8a40a5ba21a498a2d98895b4eec3f39 2025-11-03T16:15:22.3003481Z * [new tag] trunk/75f798e05b35cbdf1814673e745b9b7817bb29cd -> trunk/75f798e05b35cbdf1814673e745b9b7817bb29cd 2025-11-03T16:15:22.3003896Z * [new tag] trunk/764c54ecaeb8b0da486b009f4d4ef1fa280e2517 -> trunk/764c54ecaeb8b0da486b009f4d4ef1fa280e2517 2025-11-03T16:15:22.3004174Z * [new tag] trunk/76780b1a3d208b142f067bdd655e7d4c5615efe1 -> trunk/76780b1a3d208b142f067bdd655e7d4c5615efe1 2025-11-03T16:15:22.3004452Z * [new tag] trunk/7692fa09cdd47fdbec66efca9edf42a318e6c6e4 -> trunk/7692fa09cdd47fdbec66efca9edf42a318e6c6e4 2025-11-03T16:15:22.3004717Z * [new tag] trunk/76b2c37045e52540ec51e967aa7b6436a6b9b174 -> trunk/76b2c37045e52540ec51e967aa7b6436a6b9b174 2025-11-03T16:15:22.3004984Z * [new tag] trunk/76bb27e248750fc558ec4ea9050e09a6f171e9bc -> trunk/76bb27e248750fc558ec4ea9050e09a6f171e9bc 2025-11-03T16:15:22.3005675Z * [new tag] trunk/774abb018eb1d44c07fda9fa78549755ac7d4367 -> trunk/774abb018eb1d44c07fda9fa78549755ac7d4367 2025-11-03T16:15:22.3006092Z * [new tag] trunk/791ca80d3a2ca72a77b5dedc75bbcbb35b859cfc -> trunk/791ca80d3a2ca72a77b5dedc75bbcbb35b859cfc 2025-11-03T16:15:22.3006871Z * [new tag] trunk/797cd80b2670a51601f997f8c67387bd30440a36 -> trunk/797cd80b2670a51601f997f8c67387bd30440a36 2025-11-03T16:15:22.3007228Z * [new tag] trunk/79aa88cc5dcc55597013cbe36f0e00557b185803 -> trunk/79aa88cc5dcc55597013cbe36f0e00557b185803 2025-11-03T16:15:22.3007807Z * [new tag] trunk/79aee77381b21d41c77148e5ff84c4b351aaf144 -> trunk/79aee77381b21d41c77148e5ff84c4b351aaf144 2025-11-03T16:15:22.3008362Z * [new tag] trunk/7a0cd8ed092a62123fcc1f4c40de2669f46fe2b1 -> trunk/7a0cd8ed092a62123fcc1f4c40de2669f46fe2b1 2025-11-03T16:15:22.3008887Z * [new tag] trunk/7ae8aaf4c02e7cc14b2e3cce0c88a7333dff439c -> trunk/7ae8aaf4c02e7cc14b2e3cce0c88a7333dff439c 2025-11-03T16:15:22.3009444Z * [new tag] trunk/7c203b8420d26bd707c6052f93a4ba77dca2b2c0 -> trunk/7c203b8420d26bd707c6052f93a4ba77dca2b2c0 2025-11-03T16:15:22.3009988Z * [new tag] trunk/7ce723d21cc2a1a874f988f3459f0da834c75daa -> trunk/7ce723d21cc2a1a874f988f3459f0da834c75daa 2025-11-03T16:15:22.3010493Z * [new tag] trunk/7d16fcf2df9b240895056107a015735774cbe4b5 -> trunk/7d16fcf2df9b240895056107a015735774cbe4b5 2025-11-03T16:15:22.3011099Z * [new tag] trunk/7d39401fa07eb2f7782d8f234aa19f8b2abff481 -> trunk/7d39401fa07eb2f7782d8f234aa19f8b2abff481 2025-11-03T16:15:22.3011777Z * [new tag] trunk/7d67a41db4801632d5862e99c79b766284dfd307 -> trunk/7d67a41db4801632d5862e99c79b766284dfd307 2025-11-03T16:15:22.3012570Z * [new tag] trunk/7e3b9d105e3359e31fb252becb01d789a72dc8d9 -> trunk/7e3b9d105e3359e31fb252becb01d789a72dc8d9 2025-11-03T16:15:22.3012922Z * [new tag] trunk/7f7a28046b5927ddbd8ef95d00aa1ca76f773bb1 -> trunk/7f7a28046b5927ddbd8ef95d00aa1ca76f773bb1 2025-11-03T16:15:22.3013698Z * [new tag] trunk/80ba6e458ff2bd178da9ed62e27a61fe46e4252a -> trunk/80ba6e458ff2bd178da9ed62e27a61fe46e4252a 2025-11-03T16:15:22.3014311Z * [new tag] trunk/8101fd46d4619d0f9ef69e5829d1ee6f77289b72 -> trunk/8101fd46d4619d0f9ef69e5829d1ee6f77289b72 2025-11-03T16:15:22.3014760Z * [new tag] trunk/8110ce02a2830f562ed60607e30e2f29aed119dd -> trunk/8110ce02a2830f562ed60607e30e2f29aed119dd 2025-11-03T16:15:22.3015421Z * [new tag] trunk/81fa4a204cf268bc55e96137fa803ae4938472cd -> trunk/81fa4a204cf268bc55e96137fa803ae4938472cd 2025-11-03T16:15:22.3015963Z * [new tag] trunk/8209a0506b399b40a48716126ee22737aa4607c5 -> trunk/8209a0506b399b40a48716126ee22737aa4607c5 2025-11-03T16:15:22.3016470Z * [new tag] trunk/8221ee6db9fd319c249087ad9bb9abaa7d56e113 -> trunk/8221ee6db9fd319c249087ad9bb9abaa7d56e113 2025-11-03T16:15:22.3017381Z * [new tag] trunk/82d86bacf3c34760d6ab8a8a1eafa22d0ea259d9 -> trunk/82d86bacf3c34760d6ab8a8a1eafa22d0ea259d9 2025-11-03T16:15:22.3017628Z * [new tag] trunk/82fafb3304f8baffdb2d54578926d8bb4aba621c -> trunk/82fafb3304f8baffdb2d54578926d8bb4aba621c 2025-11-03T16:15:22.3018138Z * [new tag] trunk/82ff07c7884d478ddd5d638bebbb938e55c9bebf -> trunk/82ff07c7884d478ddd5d638bebbb938e55c9bebf 2025-11-03T16:15:22.3018706Z * [new tag] trunk/83cc38d9c1b4371d4395bda8b6861b98986f527b -> trunk/83cc38d9c1b4371d4395bda8b6861b98986f527b 2025-11-03T16:15:22.3019358Z * [new tag] trunk/840d63c12d255dac1ae3c5e442c6ea6eb87a7256 -> trunk/840d63c12d255dac1ae3c5e442c6ea6eb87a7256 2025-11-03T16:15:22.3019810Z * [new tag] trunk/8417981c96b7577f0228226ccf76d23842173144 -> trunk/8417981c96b7577f0228226ccf76d23842173144 2025-11-03T16:15:22.3020260Z * [new tag] trunk/845da9c8172b0e6e8b1e326359379dc6e9ddbddf -> trunk/845da9c8172b0e6e8b1e326359379dc6e9ddbddf 2025-11-03T16:15:22.3020767Z * [new tag] trunk/84776e13744db6d59b41a063bb8714e2bffe7a06 -> trunk/84776e13744db6d59b41a063bb8714e2bffe7a06 2025-11-03T16:15:22.3021247Z * [new tag] trunk/84a2715d341f068a26a281a252c3907bbe827d9b -> trunk/84a2715d341f068a26a281a252c3907bbe827d9b 2025-11-03T16:15:22.3021782Z * [new tag] trunk/84fe84850383fc11e7150681742bf53f11cd425e -> trunk/84fe84850383fc11e7150681742bf53f11cd425e 2025-11-03T16:15:22.3023438Z * [new tag] trunk/856a7a529872aa2d222e2491708cb458c682bb93 -> trunk/856a7a529872aa2d222e2491708cb458c682bb93 2025-11-03T16:15:22.3023685Z * [new tag] trunk/85a7c745aa5a2428a7cc640ef4b304296fe07c42 -> trunk/85a7c745aa5a2428a7cc640ef4b304296fe07c42 2025-11-03T16:15:22.3023902Z * [new tag] trunk/85b035ca9c230199d91ff86d362def01ad36dc67 -> trunk/85b035ca9c230199d91ff86d362def01ad36dc67 2025-11-03T16:15:22.3024191Z * [new tag] trunk/85b85f6c2c7e6ca98abf9d9a38410d4b5703d64a -> trunk/85b85f6c2c7e6ca98abf9d9a38410d4b5703d64a 2025-11-03T16:15:22.3024695Z * [new tag] trunk/877f126e353f537417c53bff073fc56d3faecd60 -> trunk/877f126e353f537417c53bff073fc56d3faecd60 2025-11-03T16:15:22.3025327Z * [new tag] trunk/8887a33ede05c3e77286447a8ebd82b458878943 -> trunk/8887a33ede05c3e77286447a8ebd82b458878943 2025-11-03T16:15:22.3025882Z * [new tag] trunk/895795f07cc0348ebe0b7e369d555b545d76eb9f -> trunk/895795f07cc0348ebe0b7e369d555b545d76eb9f 2025-11-03T16:15:22.3026632Z * [new tag] trunk/8aa087a29dcb5eba4cffd6a452e7f327abfd14b2 -> trunk/8aa087a29dcb5eba4cffd6a452e7f327abfd14b2 2025-11-03T16:15:22.3026908Z * [new tag] trunk/8af9ed08243222853cf8ebfc9177c5b1ea364d00 -> trunk/8af9ed08243222853cf8ebfc9177c5b1ea364d00 2025-11-03T16:15:22.3027659Z * [new tag] trunk/8b188647cfdc1355070ccd5aaa18a8060d4f67bf -> trunk/8b188647cfdc1355070ccd5aaa18a8060d4f67bf 2025-11-03T16:15:22.3027903Z * [new tag] trunk/8d4e48831e5abb6cbca8d97a2df6c47b770c303b -> trunk/8d4e48831e5abb6cbca8d97a2df6c47b770c303b 2025-11-03T16:15:22.3030418Z * [new tag] trunk/8d599045cf4102e451a9e8a9ff215d053ebbe0e8 -> trunk/8d599045cf4102e451a9e8a9ff215d053ebbe0e8 2025-11-03T16:15:22.3030660Z * [new tag] trunk/8e1e4ee8e0c43202085b47c9d2b05b2c0d13b368 -> trunk/8e1e4ee8e0c43202085b47c9d2b05b2c0d13b368 2025-11-03T16:15:22.3030869Z * [new tag] trunk/8f40a0c634a65246889b478b7f14da7a448174a9 -> trunk/8f40a0c634a65246889b478b7f14da7a448174a9 2025-11-03T16:15:22.3031111Z * [new tag] trunk/8f51556daa347041fc7ac06a832d2fdc62ca594d -> trunk/8f51556daa347041fc7ac06a832d2fdc62ca594d 2025-11-03T16:15:22.3031326Z * [new tag] trunk/904abfc2ca285910b7b1458779bd9c63619f2f7d -> trunk/904abfc2ca285910b7b1458779bd9c63619f2f7d 2025-11-03T16:15:22.3031852Z * [new tag] trunk/90519402c2006237f891289a0afdec804515aa73 -> trunk/90519402c2006237f891289a0afdec804515aa73 2025-11-03T16:15:22.3032095Z * [new tag] trunk/90b30ebf7ecd7b0da9deeb35b165de9e28e90aff -> trunk/90b30ebf7ecd7b0da9deeb35b165de9e28e90aff 2025-11-03T16:15:22.3032607Z * [new tag] trunk/90d7be35e980a39cb4044f8fefea33a6167e5733 -> trunk/90d7be35e980a39cb4044f8fefea33a6167e5733 2025-11-03T16:15:22.3033156Z * [new tag] trunk/9139368b64ebcc32c8c1190fd114c7f4733e7c6d -> trunk/9139368b64ebcc32c8c1190fd114c7f4733e7c6d 2025-11-03T16:15:22.3033727Z * [new tag] trunk/92381a5aa7c27b740d375dad2ff9ca7be3c1bc24 -> trunk/92381a5aa7c27b740d375dad2ff9ca7be3c1bc24 2025-11-03T16:15:22.3034325Z * [new tag] trunk/924482a6f63305b0f00760170ad513305700b79c -> trunk/924482a6f63305b0f00760170ad513305700b79c 2025-11-03T16:15:22.3038470Z * [new tag] trunk/9261a1fb128412201ef009d30844a2417364d73b -> trunk/9261a1fb128412201ef009d30844a2417364d73b 2025-11-03T16:15:22.3038881Z * [new tag] trunk/93a70c717a5636505da0b4ccfbca0e111c88c6a8 -> trunk/93a70c717a5636505da0b4ccfbca0e111c88c6a8 2025-11-03T16:15:22.3039232Z * [new tag] trunk/94eaeb9cb812744251cef11d147e93b00a633823 -> trunk/94eaeb9cb812744251cef11d147e93b00a633823 2025-11-03T16:15:22.3039460Z * [new tag] trunk/94f2657c4b534136aa8958bc35d44ceac5ccd60c -> trunk/94f2657c4b534136aa8958bc35d44ceac5ccd60c 2025-11-03T16:15:22.3039692Z * [new tag] trunk/9501405de686068c5d7d0e4eab21e3ef0b95cc73 -> trunk/9501405de686068c5d7d0e4eab21e3ef0b95cc73 2025-11-03T16:15:22.3039901Z * [new tag] trunk/95b55347730b071d0b79df5ddd477db9c7f91fe4 -> trunk/95b55347730b071d0b79df5ddd477db9c7f91fe4 2025-11-03T16:15:22.3040116Z * [new tag] trunk/96b61844a722d2ae8b1a2dc283a6c0060b40782e -> trunk/96b61844a722d2ae8b1a2dc283a6c0060b40782e 2025-11-03T16:15:22.3040329Z * [new tag] trunk/972030fe2ef0a7500afb7acb86a5f971c30536b9 -> trunk/972030fe2ef0a7500afb7acb86a5f971c30536b9 2025-11-03T16:15:22.3040547Z * [new tag] trunk/984e64b2cddecfe43501a7e01a1cce0c25473f54 -> trunk/984e64b2cddecfe43501a7e01a1cce0c25473f54 2025-11-03T16:15:22.3041045Z * [new tag] trunk/98d640bb113a0b6a39c590bbe36873faeb71ceb0 -> trunk/98d640bb113a0b6a39c590bbe36873faeb71ceb0 2025-11-03T16:15:22.3041263Z * [new tag] trunk/9901d4441862cc0be881f913e1a7f932f7cb0177 -> trunk/9901d4441862cc0be881f913e1a7f932f7cb0177 2025-11-03T16:15:22.3041877Z * [new tag] trunk/9940e894ea9c4374691ec49eba96db0a3159a8ae -> trunk/9940e894ea9c4374691ec49eba96db0a3159a8ae 2025-11-03T16:15:22.3042476Z * [new tag] trunk/9970fb97fff5fb5ab0f0f3acfa73558cc5c1618a -> trunk/9970fb97fff5fb5ab0f0f3acfa73558cc5c1618a 2025-11-03T16:15:22.3043000Z * [new tag] trunk/99b05d1b786dc1f2448c23617bbb80d441a5d64a -> trunk/99b05d1b786dc1f2448c23617bbb80d441a5d64a 2025-11-03T16:15:22.3045866Z * [new tag] trunk/99e07c39ec3e9cf4573da6e6737c3d0362bc6b13 -> trunk/99e07c39ec3e9cf4573da6e6737c3d0362bc6b13 2025-11-03T16:15:22.3046271Z * [new tag] trunk/9a91486e453f300804c7031d0e904b0e2af7d802 -> trunk/9a91486e453f300804c7031d0e904b0e2af7d802 2025-11-03T16:15:22.3046630Z * [new tag] trunk/9c22bbb2dce31b854e3387db77eaff501434f352 -> trunk/9c22bbb2dce31b854e3387db77eaff501434f352 2025-11-03T16:15:22.3046975Z * [new tag] trunk/9d6597b1e9f95f4477e88c5bee473c2850c2e63f -> trunk/9d6597b1e9f95f4477e88c5bee473c2850c2e63f 2025-11-03T16:15:22.3047302Z * [new tag] trunk/9ee1afbf66a6533c712956a3ba5e8f60c5875a74 -> trunk/9ee1afbf66a6533c712956a3ba5e8f60c5875a74 2025-11-03T16:15:22.3047649Z * [new tag] trunk/9f9dbe0a9aa5e5c985f36ad6cc736b3fb3506056 -> trunk/9f9dbe0a9aa5e5c985f36ad6cc736b3fb3506056 2025-11-03T16:15:22.3048017Z * [new tag] trunk/a04edcb27acc6e1dd0fc27233738e6dec718b109 -> trunk/a04edcb27acc6e1dd0fc27233738e6dec718b109 2025-11-03T16:15:22.3048259Z * [new tag] trunk/a076b4d7ac09b7be59915a088b348bcbd952c8bd -> trunk/a076b4d7ac09b7be59915a088b348bcbd952c8bd 2025-11-03T16:15:22.3048480Z * [new tag] trunk/a186aa8d6cc197054deba309605f4b75ca59899c -> trunk/a186aa8d6cc197054deba309605f4b75ca59899c 2025-11-03T16:15:22.3048824Z * [new tag] trunk/a19e92d43355489c8d33325321e688a6e6182267 -> trunk/a19e92d43355489c8d33325321e688a6e6182267 2025-11-03T16:15:22.3049160Z * [new tag] trunk/a1eb6b55387b6ae21cf9bcad76a471a2820fa1b0 -> trunk/a1eb6b55387b6ae21cf9bcad76a471a2820fa1b0 2025-11-03T16:15:22.3049574Z * [new tag] trunk/a25818cf7ee2c0ed5c862dff214dc46a30211671 -> trunk/a25818cf7ee2c0ed5c862dff214dc46a30211671 2025-11-03T16:15:22.3050970Z * [new tag] trunk/a2da69385abb4800732e698afacdddd78e9993aa -> trunk/a2da69385abb4800732e698afacdddd78e9993aa 2025-11-03T16:15:22.3051372Z * [new tag] trunk/a3fe1825aa4d95707875ffa8f15fd00c3b94977d -> trunk/a3fe1825aa4d95707875ffa8f15fd00c3b94977d 2025-11-03T16:15:22.3051727Z * [new tag] trunk/a4077b568f891f8cad948cc70cbb14043791d762 -> trunk/a4077b568f891f8cad948cc70cbb14043791d762 2025-11-03T16:15:22.3052079Z * [new tag] trunk/a4a0378e6bb8afd774e4e253304cf3bf9d7e188a -> trunk/a4a0378e6bb8afd774e4e253304cf3bf9d7e188a 2025-11-03T16:15:22.3052417Z * [new tag] trunk/a51f8772873bc91de8aad1ce865fe8d98a58aec7 -> trunk/a51f8772873bc91de8aad1ce865fe8d98a58aec7 2025-11-03T16:15:22.3052776Z * [new tag] trunk/a5335263d32b5be2b2647661334d81225c3cc3fc -> trunk/a5335263d32b5be2b2647661334d81225c3cc3fc 2025-11-03T16:15:22.3053457Z * [new tag] trunk/a553ea9ea4e71b66a58a1c3c37f18d67ffa4ecd3 -> trunk/a553ea9ea4e71b66a58a1c3c37f18d67ffa4ecd3 2025-11-03T16:15:22.3054072Z * [new tag] trunk/a5c3c08d1099f96548398a637463efb0cbe115ae -> trunk/a5c3c08d1099f96548398a637463efb0cbe115ae 2025-11-03T16:15:22.3054366Z * [new tag] trunk/a5f00077fc9623c762c755c926b7eee60eb0ec3f -> trunk/a5f00077fc9623c762c755c926b7eee60eb0ec3f 2025-11-03T16:15:22.3054861Z * [new tag] trunk/a663eb9c80e92190832b3dd616f138f938f51525 -> trunk/a663eb9c80e92190832b3dd616f138f938f51525 2025-11-03T16:15:22.3055477Z * [new tag] trunk/a6b1ef17173f56ba93ac97ff4384fa4060b5e41e -> trunk/a6b1ef17173f56ba93ac97ff4384fa4060b5e41e 2025-11-03T16:15:22.3055988Z * [new tag] trunk/a76b59cc45a17b728d9da6f937c8a9ab537c657f -> trunk/a76b59cc45a17b728d9da6f937c8a9ab537c657f 2025-11-03T16:15:22.3056539Z * [new tag] trunk/a77f5d9a005a789804fc4d5064036490db5f48fb -> trunk/a77f5d9a005a789804fc4d5064036490db5f48fb 2025-11-03T16:15:22.3057315Z * [new tag] trunk/a7fd0b40018da3c692f74257fc9b1d8cc5c410ff -> trunk/a7fd0b40018da3c692f74257fc9b1d8cc5c410ff 2025-11-03T16:15:22.3057618Z * [new tag] trunk/a988510c3396a668e92801c61b93f7b06219ad0e -> trunk/a988510c3396a668e92801c61b93f7b06219ad0e 2025-11-03T16:15:22.3058233Z * [new tag] trunk/a9b29caeae77480abaee14c89cff19f2c23921a1 -> trunk/a9b29caeae77480abaee14c89cff19f2c23921a1 2025-11-03T16:15:22.3058759Z * [new tag] trunk/aa9c96af041b26c9c55adac490f3449b98f27d06 -> trunk/aa9c96af041b26c9c55adac490f3449b98f27d06 2025-11-03T16:15:22.3059298Z * [new tag] trunk/aab27b051a5e24e71931ef0386f9084d51347211 -> trunk/aab27b051a5e24e71931ef0386f9084d51347211 2025-11-03T16:15:22.3059834Z * [new tag] trunk/ac841267a197e102e1800786b3f6bfcd068b3d85 -> trunk/ac841267a197e102e1800786b3f6bfcd068b3d85 2025-11-03T16:15:22.3061325Z * [new tag] trunk/acd936cc1a53b3c97f1265b6e40e2c09ba0a182d -> trunk/acd936cc1a53b3c97f1265b6e40e2c09ba0a182d 2025-11-03T16:15:22.3061584Z * [new tag] trunk/ad02bd13dfa017f69def846b265a566c4ec5cb3f -> trunk/ad02bd13dfa017f69def846b265a566c4ec5cb3f 2025-11-03T16:15:22.3062099Z * [new tag] trunk/ad3a56ab9857905cdc8f62fb635d578982797c73 -> trunk/ad3a56ab9857905cdc8f62fb635d578982797c73 2025-11-03T16:15:22.3062352Z * [new tag] trunk/ad559072db0cc68aad277380d20bacfdf0d944ac -> trunk/ad559072db0cc68aad277380d20bacfdf0d944ac 2025-11-03T16:15:22.3062812Z * [new tag] trunk/add37bacda45d7de852dedee652346f2857df6fc -> trunk/add37bacda45d7de852dedee652346f2857df6fc 2025-11-03T16:15:22.3063512Z * [new tag] trunk/adedf26e21d1ce835750c99464b3ffa0f6b38fd0 -> trunk/adedf26e21d1ce835750c99464b3ffa0f6b38fd0 2025-11-03T16:15:22.3064237Z * [new tag] trunk/ae038f871b3cb91c36df3a9d4b0ffb4f35d20482 -> trunk/ae038f871b3cb91c36df3a9d4b0ffb4f35d20482 2025-11-03T16:15:22.3064540Z * [new tag] trunk/afaaaa314cc9358a10e9b1986642d49c00773560 -> trunk/afaaaa314cc9358a10e9b1986642d49c00773560 2025-11-03T16:15:22.3065009Z * [new tag] trunk/b060e5c131e9cf43ba6e9b5bda2cb5789c61de47 -> trunk/b060e5c131e9cf43ba6e9b5bda2cb5789c61de47 2025-11-03T16:15:22.3066080Z * [new tag] trunk/b0831930edaa732dd73dc81223fc0a9f5fe6b956 -> trunk/b0831930edaa732dd73dc81223fc0a9f5fe6b956 2025-11-03T16:15:22.3066454Z * [new tag] trunk/b09fb481e0fb00dc475f218e9b14660161b40a94 -> trunk/b09fb481e0fb00dc475f218e9b14660161b40a94 2025-11-03T16:15:22.3066776Z * [new tag] trunk/b2a0f90501dd3a16a6ccaf4c49e1c10f6df4ce1d -> trunk/b2a0f90501dd3a16a6ccaf4c49e1c10f6df4ce1d 2025-11-03T16:15:22.3067294Z * [new tag] trunk/b33762bd2f5f1b115f049d83bf553b08d1ffc97f -> trunk/b33762bd2f5f1b115f049d83bf553b08d1ffc97f 2025-11-03T16:15:22.3067918Z * [new tag] trunk/b3861ac8e70a2334278c17f834880e2d45bbe7a9 -> trunk/b3861ac8e70a2334278c17f834880e2d45bbe7a9 2025-11-03T16:15:22.3068519Z * [new tag] trunk/b4403bfc62ca97eec554cdf815baab1fe93057d9 -> trunk/b4403bfc62ca97eec554cdf815baab1fe93057d9 2025-11-03T16:15:22.3069100Z * [new tag] trunk/b44423bbb43860c1e340cbebc9d101dc18031ecb -> trunk/b44423bbb43860c1e340cbebc9d101dc18031ecb 2025-11-03T16:15:22.3069644Z * [new tag] trunk/b470e59c382649557c49d3c31b8dec1b036eb329 -> trunk/b470e59c382649557c49d3c31b8dec1b036eb329 2025-11-03T16:15:22.3070264Z * [new tag] trunk/b5189e269e899a6ad9ddc4e2ce6f33029290af82 -> trunk/b5189e269e899a6ad9ddc4e2ce6f33029290af82 2025-11-03T16:15:22.3070851Z * [new tag] trunk/b71966f67b2b688f9d71f1602bd3c52d413dacef -> trunk/b71966f67b2b688f9d71f1602bd3c52d413dacef 2025-11-03T16:15:22.3071270Z * [new tag] trunk/b7d348a9074af70f34848298a4e7a7985b197742 -> trunk/b7d348a9074af70f34848298a4e7a7985b197742 2025-11-03T16:15:22.3071925Z * [new tag] trunk/b903018c26217edca67c9023b104027aea377b83 -> trunk/b903018c26217edca67c9023b104027aea377b83 2025-11-03T16:15:22.3072548Z * [new tag] trunk/b939de26d115d33d3cbc17dcf49dbdc1a84e9727 -> trunk/b939de26d115d33d3cbc17dcf49dbdc1a84e9727 2025-11-03T16:15:22.3073097Z * [new tag] trunk/b9bcb37f40e72dc4c2fc1d9281175409fe474194 -> trunk/b9bcb37f40e72dc4c2fc1d9281175409fe474194 2025-11-03T16:15:22.3074369Z * [new tag] trunk/ba71e9ca9ac21e6570b20353d0a67089ad32e442 -> trunk/ba71e9ca9ac21e6570b20353d0a67089ad32e442 2025-11-03T16:15:22.3074606Z * [new tag] trunk/bb54296258854897058e8f158afa8efedc4fcf18 -> trunk/bb54296258854897058e8f158afa8efedc4fcf18 2025-11-03T16:15:22.3075143Z * [new tag] trunk/bbb7d2270b6f647b97f8a9c1d4e69ddaf48ef661 -> trunk/bbb7d2270b6f647b97f8a9c1d4e69ddaf48ef661 2025-11-03T16:15:22.3075621Z * [new tag] trunk/bc03d7c97462abfd45fc1b7b73d5c7acbbd2c185 -> trunk/bc03d7c97462abfd45fc1b7b73d5c7acbbd2c185 2025-11-03T16:15:22.3076190Z * [new tag] trunk/bc5111cd8dc5167cbcbcb2c0f23e53e61a6e8ea5 -> trunk/bc5111cd8dc5167cbcbcb2c0f23e53e61a6e8ea5 2025-11-03T16:15:22.3076748Z * [new tag] trunk/be283297100ab86123e74b7a8372995d32b140c8 -> trunk/be283297100ab86123e74b7a8372995d32b140c8 2025-11-03T16:15:22.3077572Z * [new tag] trunk/bea89d60603ca33f1903e57db5c734a67535c26d -> trunk/bea89d60603ca33f1903e57db5c734a67535c26d 2025-11-03T16:15:22.3077938Z * [new tag] trunk/bebabd7fce29ea49b9269aeaa9fe3f34a3e1127e -> trunk/bebabd7fce29ea49b9269aeaa9fe3f34a3e1127e 2025-11-03T16:15:22.3078476Z * [new tag] trunk/bfb47ec50e09dc2921ba47fedf84a1e28ee991a7 -> trunk/bfb47ec50e09dc2921ba47fedf84a1e28ee991a7 2025-11-03T16:15:22.3078858Z * [new tag] trunk/bfc2050db975e589795cd3eceaed2e83bf89ad35 -> trunk/bfc2050db975e589795cd3eceaed2e83bf89ad35 2025-11-03T16:15:22.3079429Z * [new tag] trunk/c01636e1bc7338eecbcfd2c17fd3fa2cad836983 -> trunk/c01636e1bc7338eecbcfd2c17fd3fa2cad836983 2025-11-03T16:15:22.3079985Z * [new tag] trunk/c0bbda37e8d74734b64421762eb365e620f258c1 -> trunk/c0bbda37e8d74734b64421762eb365e620f258c1 2025-11-03T16:15:22.3081294Z * [new tag] trunk/c201a1cab1439bfcfaff7e4fdab0d6eeeb880a70 -> trunk/c201a1cab1439bfcfaff7e4fdab0d6eeeb880a70 2025-11-03T16:15:22.3081704Z * [new tag] trunk/c2e3cc7aedb2e7d89443225c7cccd08a0f8a3587 -> trunk/c2e3cc7aedb2e7d89443225c7cccd08a0f8a3587 2025-11-03T16:15:22.3082069Z * [new tag] trunk/c3047938a01c844c752e5462b05d180a04d438fd -> trunk/c3047938a01c844c752e5462b05d180a04d438fd 2025-11-03T16:15:22.3082381Z * [new tag] trunk/c37802a8c4388160faa65f8bc9c29d98b4d2fb3c -> trunk/c37802a8c4388160faa65f8bc9c29d98b4d2fb3c 2025-11-03T16:15:22.3082812Z * [new tag] trunk/c3b71d54995a16fd09c18a5da5d598a23388bbd6 -> trunk/c3b71d54995a16fd09c18a5da5d598a23388bbd6 2025-11-03T16:15:22.3083338Z * [new tag] trunk/c3d205d5986824e72df353c9bf112dc9b2a3612a -> trunk/c3d205d5986824e72df353c9bf112dc9b2a3612a 2025-11-03T16:15:22.3083751Z * [new tag] trunk/c3dc0c7089f3204c36961acc227a3ff503530094 -> trunk/c3dc0c7089f3204c36961acc227a3ff503530094 2025-11-03T16:15:22.3084591Z * [new tag] trunk/c54e2c5b4138cb85412a74189d1f7f771ad1116e -> trunk/c54e2c5b4138cb85412a74189d1f7f771ad1116e 2025-11-03T16:15:22.3084913Z * [new tag] trunk/c5701d0ab5f55b23243dd437f57e856970c45bb6 -> trunk/c5701d0ab5f55b23243dd437f57e856970c45bb6 2025-11-03T16:15:22.3085596Z * [new tag] trunk/c58d0ad85dfbf026d5f21ee739c139407dc20e54 -> trunk/c58d0ad85dfbf026d5f21ee739c139407dc20e54 2025-11-03T16:15:22.3086160Z * [new tag] trunk/c594950e867e420a6086190a88e4b8fbd687af91 -> trunk/c594950e867e420a6086190a88e4b8fbd687af91 2025-11-03T16:15:22.3087473Z * [new tag] trunk/c6a02eae5b77758ee7ee0b1dfbcf20a8341914a7 -> trunk/c6a02eae5b77758ee7ee0b1dfbcf20a8341914a7 2025-11-03T16:15:22.3087735Z * [new tag] trunk/c8adc08b3b19b838dcdc71f2055600f420e6a91b -> trunk/c8adc08b3b19b838dcdc71f2055600f420e6a91b 2025-11-03T16:15:22.3088016Z * [new tag] trunk/c9eabadc5e964678556a6bfcde455a5e11dc0785 -> trunk/c9eabadc5e964678556a6bfcde455a5e11dc0785 2025-11-03T16:15:22.3088677Z * [new tag] trunk/cb6966704cf70ebac6d2b36b65e9a75e11c29e71 -> trunk/cb6966704cf70ebac6d2b36b65e9a75e11c29e71 2025-11-03T16:15:22.3089030Z * [new tag] trunk/cde81e92b95eee9af2879c9c75f7b03699ca72ad -> trunk/cde81e92b95eee9af2879c9c75f7b03699ca72ad 2025-11-03T16:15:22.3089495Z * [new tag] trunk/cee03634dab2dbf259d84bc389bdea80de0f80ac -> trunk/cee03634dab2dbf259d84bc389bdea80de0f80ac 2025-11-03T16:15:22.3090129Z * [new tag] trunk/cf7756da385e03c169f7e6a5b5ff2c0e29b2814a -> trunk/cf7756da385e03c169f7e6a5b5ff2c0e29b2814a 2025-11-03T16:15:22.3090681Z * [new tag] trunk/cf9a834f39d6c3b769381cc1321bcb5c54132415 -> trunk/cf9a834f39d6c3b769381cc1321bcb5c54132415 2025-11-03T16:15:22.3091198Z * [new tag] trunk/d049ed2cb1619c44279cb716b8a1d94e4df3b372 -> trunk/d049ed2cb1619c44279cb716b8a1d94e4df3b372 2025-11-03T16:15:22.3091807Z * [new tag] trunk/d17790072381f0e76e177ae4093504a6908a5a4d -> trunk/d17790072381f0e76e177ae4093504a6908a5a4d 2025-11-03T16:15:22.3092292Z * [new tag] trunk/d1a6e006e090b1365e6653853982f175cca452e8 -> trunk/d1a6e006e090b1365e6653853982f175cca452e8 2025-11-03T16:15:22.3092813Z * [new tag] trunk/d2be06f673ff11e59b234156dccaad2dc22c7746 -> trunk/d2be06f673ff11e59b234156dccaad2dc22c7746 2025-11-03T16:15:22.3094415Z * [new tag] trunk/d2eff5d454ab2cb0a5ccdfb5eb6e7d6dcc75e097 -> trunk/d2eff5d454ab2cb0a5ccdfb5eb6e7d6dcc75e097 2025-11-03T16:15:22.3094669Z * [new tag] trunk/d3be06cbdca4b2a9cd8ae35926b3f20b1bed1fe2 -> trunk/d3be06cbdca4b2a9cd8ae35926b3f20b1bed1fe2 2025-11-03T16:15:22.3095774Z * [new tag] trunk/d3e511f07c0b1fc6df22347aec7f8e700bdb0d24 -> trunk/d3e511f07c0b1fc6df22347aec7f8e700bdb0d24 2025-11-03T16:15:22.3096028Z * [new tag] trunk/d401e4e70a1dd19733d3e3c84affc4884605efb4 -> trunk/d401e4e70a1dd19733d3e3c84affc4884605efb4 2025-11-03T16:15:22.3096466Z * [new tag] trunk/d46d8d6f54b15ded4f2483c7bde31be124281ab8 -> trunk/d46d8d6f54b15ded4f2483c7bde31be124281ab8 2025-11-03T16:15:22.3096960Z * [new tag] trunk/d6d6fa26f540c10c57ac80547a9475e9f4c201f2 -> trunk/d6d6fa26f540c10c57ac80547a9475e9f4c201f2 2025-11-03T16:15:22.3097486Z * [new tag] trunk/d7040e6d7515cea485824d2b810bea94e5958dea -> trunk/d7040e6d7515cea485824d2b810bea94e5958dea 2025-11-03T16:15:22.3098328Z * [new tag] trunk/d80ae738c9c1810b30a88233bdcd34cae16c5c6e -> trunk/d80ae738c9c1810b30a88233bdcd34cae16c5c6e 2025-11-03T16:15:22.3098611Z * [new tag] trunk/d8283a317a335c628c4541c6a60e102fdead57b7 -> trunk/d8283a317a335c628c4541c6a60e102fdead57b7 2025-11-03T16:15:22.3101496Z * [new tag] trunk/d9483d4c8d3f220ad505afd05606e068184896a4 -> trunk/d9483d4c8d3f220ad505afd05606e068184896a4 2025-11-03T16:15:22.3101909Z * [new tag] trunk/d962bed15798961ffd10e64068f86a2cec411ffb -> trunk/d962bed15798961ffd10e64068f86a2cec411ffb 2025-11-03T16:15:22.3102258Z * [new tag] trunk/d97144d31ef42e762ce65f467cbaf065edb241cc -> trunk/d97144d31ef42e762ce65f467cbaf065edb241cc 2025-11-03T16:15:22.3102763Z * [new tag] trunk/dc011d3203b24ebf1f78ab8dd64766205e64d8ec -> trunk/dc011d3203b24ebf1f78ab8dd64766205e64d8ec 2025-11-03T16:15:22.3103457Z * [new tag] trunk/dd1fe7c22f8eee3984a22663636b0d1403b079ba -> trunk/dd1fe7c22f8eee3984a22663636b0d1403b079ba 2025-11-03T16:15:22.3103723Z * [new tag] trunk/deb776319b12cad7fffb5cf9d8851a50e2b0e9ea -> trunk/deb776319b12cad7fffb5cf9d8851a50e2b0e9ea 2025-11-03T16:15:22.3103947Z * [new tag] trunk/defac66e39c60f867795346ded171366ed682ee3 -> trunk/defac66e39c60f867795346ded171366ed682ee3 2025-11-03T16:15:22.3104191Z * [new tag] trunk/df71b7072799c451a008cb36142dfdb1487f0d5e -> trunk/df71b7072799c451a008cb36142dfdb1487f0d5e 2025-11-03T16:15:22.3104429Z * [new tag] trunk/dfebdcab86acbaa0eaa996b47595e5f27a66492e -> trunk/dfebdcab86acbaa0eaa996b47595e5f27a66492e 2025-11-03T16:15:22.3104822Z * [new tag] trunk/e0604d3170f1123ebacbced66be400c7f162621f -> trunk/e0604d3170f1123ebacbced66be400c7f162621f 2025-11-03T16:15:22.3105170Z * [new tag] trunk/e0791fc11dc0024a828495985898b29120dcc4c1 -> trunk/e0791fc11dc0024a828495985898b29120dcc4c1 2025-11-03T16:15:22.3105955Z * [new tag] trunk/e0ca3049c0c7726747094f47f80cadcfad51320d -> trunk/e0ca3049c0c7726747094f47f80cadcfad51320d 2025-11-03T16:15:22.3106201Z * [new tag] trunk/e105a475756ae6dd45c8134bd4583afc57ee416e -> trunk/e105a475756ae6dd45c8134bd4583afc57ee416e 2025-11-03T16:15:22.3106430Z * [new tag] trunk/e137cd0a1046ead2ca9d631f427655adced606c2 -> trunk/e137cd0a1046ead2ca9d631f427655adced606c2 2025-11-03T16:15:22.3107114Z * [new tag] trunk/e1d011d6eb571cd98ec7c7ed8e8b518a5463ec97 -> trunk/e1d011d6eb571cd98ec7c7ed8e8b518a5463ec97 2025-11-03T16:15:22.3107354Z * [new tag] trunk/e214af6ae8e613485ac3dac3e2094cf53dfc3c1d -> trunk/e214af6ae8e613485ac3dac3e2094cf53dfc3c1d 2025-11-03T16:15:22.3107937Z * [new tag] trunk/e2dc32f4baf853e6171777aed530fc83bcdb35ae -> trunk/e2dc32f4baf853e6171777aed530fc83bcdb35ae 2025-11-03T16:15:22.3108389Z * [new tag] trunk/e380028a51c38de80b8888d027c63e290a773651 -> trunk/e380028a51c38de80b8888d027c63e290a773651 2025-11-03T16:15:22.3109091Z * [new tag] trunk/e3ae0594d16134632ff587c9ab400d4148c83e9f -> trunk/e3ae0594d16134632ff587c9ab400d4148c83e9f 2025-11-03T16:15:22.3109312Z * [new tag] trunk/e3e93c7107830c13f4139c3a62fda62c6b84bbf5 -> trunk/e3e93c7107830c13f4139c3a62fda62c6b84bbf5 2025-11-03T16:15:22.3113454Z * [new tag] trunk/e4043884c734a3947f4c8da0cfbba43fc6319f3d -> trunk/e4043884c734a3947f4c8da0cfbba43fc6319f3d 2025-11-03T16:15:22.3113745Z * [new tag] trunk/e471800dceb8a6592562fc4fc96a5e80bc494c0a -> trunk/e471800dceb8a6592562fc4fc96a5e80bc494c0a 2025-11-03T16:15:22.3114086Z * [new tag] trunk/e83be7042e809ef51c67f53d83ac45794447c73f -> trunk/e83be7042e809ef51c67f53d83ac45794447c73f 2025-11-03T16:15:22.3114330Z * [new tag] trunk/e8d887ae3fceb97283bc2d751621b9615524b4b7 -> trunk/e8d887ae3fceb97283bc2d751621b9615524b4b7 2025-11-03T16:15:22.3114573Z * [new tag] trunk/e8fadba28cc225782ddce1bf0ef8315ee1743761 -> trunk/e8fadba28cc225782ddce1bf0ef8315ee1743761 2025-11-03T16:15:22.3114822Z * [new tag] trunk/e95920e3e6f9d8218c6aed2b8b6ce9b838a49104 -> trunk/e95920e3e6f9d8218c6aed2b8b6ce9b838a49104 2025-11-03T16:15:22.3115071Z * [new tag] trunk/ea698e8bfcd2ec1820b3ba90b9c47416a33348de -> trunk/ea698e8bfcd2ec1820b3ba90b9c47416a33348de 2025-11-03T16:15:22.3120156Z * [new tag] trunk/eae701cad03feb87908107cb5246d9e13a9d426e -> trunk/eae701cad03feb87908107cb5246d9e13a9d426e 2025-11-03T16:15:22.3125341Z * [new tag] trunk/eb2bad5bb54df8a2563f2619819e9e689e0ba10e -> trunk/eb2bad5bb54df8a2563f2619819e9e689e0ba10e 2025-11-03T16:15:22.3127139Z * [new tag] trunk/ebb2b2e894a4ede8efc5f5fff068d4ac9972f77a -> trunk/ebb2b2e894a4ede8efc5f5fff068d4ac9972f77a 2025-11-03T16:15:22.3127490Z * [new tag] trunk/ee7434be822cf6e75b4566d8159f550ee233d8ae -> trunk/ee7434be822cf6e75b4566d8159f550ee233d8ae 2025-11-03T16:15:22.3130706Z * [new tag] trunk/ef8d97efcf05e1f3a05ad5ce3444e3af42481505 -> trunk/ef8d97efcf05e1f3a05ad5ce3444e3af42481505 2025-11-03T16:15:22.3131122Z * [new tag] trunk/f013e804c876a34cbc86a6a475597e3e29812883 -> trunk/f013e804c876a34cbc86a6a475597e3e29812883 2025-11-03T16:15:22.3131487Z * [new tag] trunk/f02708c2be66c7cf0edec52663de69f66ba4b1df -> trunk/f02708c2be66c7cf0edec52663de69f66ba4b1df 2025-11-03T16:15:22.3131756Z * [new tag] trunk/f0745ddb11ab66a8bb4de8031bd7e9dccba19379 -> trunk/f0745ddb11ab66a8bb4de8031bd7e9dccba19379 2025-11-03T16:15:22.3132359Z * [new tag] trunk/f167fd09fabf3f5be63cd31451f29d49796c14e8 -> trunk/f167fd09fabf3f5be63cd31451f29d49796c14e8 2025-11-03T16:15:22.3132645Z * [new tag] trunk/f1a34407152302812bf633360cd29e9fba8de71e -> trunk/f1a34407152302812bf633360cd29e9fba8de71e 2025-11-03T16:15:22.3132872Z * [new tag] trunk/f1af679270392c83e03808c8af5e2cbe3cdf16ce -> trunk/f1af679270392c83e03808c8af5e2cbe3cdf16ce 2025-11-03T16:15:22.3133112Z * [new tag] trunk/f1e4c42b6ef3d3cea08ab3babb693e3ce42cf08b -> trunk/f1e4c42b6ef3d3cea08ab3babb693e3ce42cf08b 2025-11-03T16:15:22.3133326Z * [new tag] trunk/f20bf778746a7662a9bdbd2fa49d326772ca90ed -> trunk/f20bf778746a7662a9bdbd2fa49d326772ca90ed 2025-11-03T16:15:22.3133543Z * [new tag] trunk/f2450798cd6f30b811f6d8256da0c271cad2285c -> trunk/f2450798cd6f30b811f6d8256da0c271cad2285c 2025-11-03T16:15:22.3133950Z * [new tag] trunk/f2c81635c85bf8d1c523d6b17f8016ad78ea02e2 -> trunk/f2c81635c85bf8d1c523d6b17f8016ad78ea02e2 2025-11-03T16:15:22.3134180Z * [new tag] trunk/f33abae6957b8ef56268ce22e660d3ed3111c7c3 -> trunk/f33abae6957b8ef56268ce22e660d3ed3111c7c3 2025-11-03T16:15:22.3134388Z * [new tag] trunk/f36f372acc28062e0988d84699c62689b0d89a6e -> trunk/f36f372acc28062e0988d84699c62689b0d89a6e 2025-11-03T16:15:22.3134602Z * [new tag] trunk/f452edd782fb7342e2a5740cea9b78135e714858 -> trunk/f452edd782fb7342e2a5740cea9b78135e714858 2025-11-03T16:15:22.3134826Z * [new tag] trunk/f4d05feb7a5f9191509e1e539ea210fcbe538b32 -> trunk/f4d05feb7a5f9191509e1e539ea210fcbe538b32 2025-11-03T16:15:22.3135039Z * [new tag] trunk/f5543e3741bbfba4579afc3b49d0720f2eee7e4d -> trunk/f5543e3741bbfba4579afc3b49d0720f2eee7e4d 2025-11-03T16:15:22.3135261Z * [new tag] trunk/f5cb9a4c68d9271c58ef4d3257210984b8e85099 -> trunk/f5cb9a4c68d9271c58ef4d3257210984b8e85099 2025-11-03T16:15:22.3135467Z * [new tag] trunk/f60751024ecac57840a2d25132d0c1562351e66e -> trunk/f60751024ecac57840a2d25132d0c1562351e66e 2025-11-03T16:15:22.3135693Z * [new tag] trunk/f6951cb8ea1ef96a33627734eb68b5dc7a04b6ed -> trunk/f6951cb8ea1ef96a33627734eb68b5dc7a04b6ed 2025-11-03T16:15:22.3135913Z * [new tag] trunk/f89a7e9fe80fb0f74af52bad892e1afafa57be9c -> trunk/f89a7e9fe80fb0f74af52bad892e1afafa57be9c 2025-11-03T16:15:22.3136134Z * [new tag] trunk/f8b4c002940d96ba2afc14ac4cf0b5277776bc02 -> trunk/f8b4c002940d96ba2afc14ac4cf0b5277776bc02 2025-11-03T16:15:22.3136341Z * [new tag] trunk/f911d64750efb22097eef5505552a7669a0c53de -> trunk/f911d64750efb22097eef5505552a7669a0c53de 2025-11-03T16:15:22.3136550Z * [new tag] trunk/f91899ca6c15d611c453591527e1be8b43e5d4ea -> trunk/f91899ca6c15d611c453591527e1be8b43e5d4ea 2025-11-03T16:15:22.3136773Z * [new tag] trunk/f93ea7dab190fa0953b7a4921b76993cadafc7c9 -> trunk/f93ea7dab190fa0953b7a4921b76993cadafc7c9 2025-11-03T16:15:22.3136985Z * [new tag] trunk/fa4cb918462d3cbe7cf48db7c1498824c1ca1cc4 -> trunk/fa4cb918462d3cbe7cf48db7c1498824c1ca1cc4 2025-11-03T16:15:22.3137248Z * [new tag] trunk/fa560e1158b496d1c37ca2a2443a0b7f5a08dad5 -> trunk/fa560e1158b496d1c37ca2a2443a0b7f5a08dad5 2025-11-03T16:15:22.3137461Z * [new tag] trunk/fa8e073a4e17f12e7e4bb0b26471179d2ca57471 -> trunk/fa8e073a4e17f12e7e4bb0b26471179d2ca57471 2025-11-03T16:15:22.3137684Z * [new tag] trunk/fb545fb0686c94cc017c15ca7e342996b49edb81 -> trunk/fb545fb0686c94cc017c15ca7e342996b49edb81 2025-11-03T16:15:22.3137902Z * [new tag] trunk/fc540cefd498f1001a5fc7a4f187080628b27839 -> trunk/fc540cefd498f1001a5fc7a4f187080628b27839 2025-11-03T16:15:22.3138146Z * [new tag] trunk/fc8ac1216c61d8736f45a58825abf58e7d51e088 -> trunk/fc8ac1216c61d8736f45a58825abf58e7d51e088 2025-11-03T16:15:22.3138359Z * [new tag] trunk/fcc10635660ceb619f78d06eaaa8b3c0bdd52ab8 -> trunk/fcc10635660ceb619f78d06eaaa8b3c0bdd52ab8 2025-11-03T16:15:22.3138576Z * [new tag] trunk/fcd5f8c352b5b75bd32e57fa044ec5df095032da -> trunk/fcd5f8c352b5b75bd32e57fa044ec5df095032da 2025-11-03T16:15:22.3138805Z * [new tag] trunk/fd5da81fdd76c0a78ef935d077a93b48b78fca3a -> trunk/fd5da81fdd76c0a78ef935d077a93b48b78fca3a 2025-11-03T16:15:22.3139024Z * [new tag] trunk/fd68d409ada709450ced3030bde89ec662a3f7b7 -> trunk/fd68d409ada709450ced3030bde89ec662a3f7b7 2025-11-03T16:15:22.3139244Z * [new tag] trunk/fea819ed080175f9a93205cc9074022bfa486325 -> trunk/fea819ed080175f9a93205cc9074022bfa486325 2025-11-03T16:15:22.3139464Z * [new tag] trunk/fee1ac927d66fce67f862311292dcc325b8def44 -> trunk/fee1ac927d66fce67f862311292dcc325b8def44 2025-11-03T16:15:22.3139727Z * [new tag] trunk/fee7624bd66e64f575c1396fcaaa54cabce49ca4 -> trunk/fee7624bd66e64f575c1396fcaaa54cabce49ca4 2025-11-03T16:15:22.3139951Z * [new tag] trunk/fefb546b9100614da1fa703e12651bc115059f02 -> trunk/fefb546b9100614da1fa703e12651bc115059f02 2025-11-03T16:15:22.3140185Z * [new tag] trunk/ff46d5a79bec09d9ae334d325f5f8cc89306b536 -> trunk/ff46d5a79bec09d9ae334d325f5f8cc89306b536 2025-11-03T16:15:22.3140405Z * [new tag] trunk/ffaa6578b7a641b1ac1fcf451f04cf5e1212c9fe -> trunk/ffaa6578b7a641b1ac1fcf451f04cf5e1212c9fe 2025-11-03T16:15:22.3140513Z * [new tag] v0.1.1 -> v0.1.1 2025-11-03T16:15:22.3140621Z * [new tag] v0.1.10 -> v0.1.10 2025-11-03T16:15:22.3140887Z * [new tag] v0.1.11 -> v0.1.11 2025-11-03T16:15:22.3141131Z * [new tag] v0.1.12 -> v0.1.12 2025-11-03T16:15:22.3142251Z * [new tag] v0.1.2 -> v0.1.2 2025-11-03T16:15:22.3142379Z * [new tag] v0.1.3 -> v0.1.3 2025-11-03T16:15:22.3142606Z * [new tag] v0.1.4 -> v0.1.4 2025-11-03T16:15:22.3143993Z * [new tag] v0.1.5 -> v0.1.5 2025-11-03T16:15:22.3144290Z * [new tag] v0.1.6 -> v0.1.6 2025-11-03T16:15:22.3144399Z * [new tag] v0.1.7 -> v0.1.7 2025-11-03T16:15:22.3144500Z * [new tag] v0.1.8 -> v0.1.8 2025-11-03T16:15:22.3146794Z * [new tag] v0.1.9 -> v0.1.9 2025-11-03T16:15:22.3147080Z * [new tag] v0.2.0 -> v0.2.0 2025-11-03T16:15:22.3147187Z * [new tag] v0.3.0 -> v0.3.0 2025-11-03T16:15:22.3147293Z * [new tag] v0.3.1 -> v0.3.1 2025-11-03T16:15:22.3147554Z * [new tag] v0.4.0 -> v0.4.0 2025-11-03T16:15:22.3148776Z * [new tag] v0.4.1 -> v0.4.1 2025-11-03T16:15:22.3148900Z * [new tag] v1.0.0 -> v1.0.0 2025-11-03T16:15:22.3149230Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-11-03T16:15:22.3149714Z * [new tag] v1.0.1 -> v1.0.1 2025-11-03T16:15:22.3150592Z * [new tag] v1.0rc0 -> v1.0rc0 2025-11-03T16:15:22.3151193Z * [new tag] v1.0rc1 -> v1.0rc1 2025-11-03T16:15:22.3151345Z * [new tag] v1.1.0 -> v1.1.0 2025-11-03T16:15:22.3151740Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-11-03T16:15:22.3152859Z * [new tag] v1.10.0 -> v1.10.0 2025-11-03T16:15:22.3153116Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-11-03T16:15:22.3154162Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-11-03T16:15:22.3154366Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-11-03T16:15:22.3154712Z * [new tag] v1.10.1 -> v1.10.1 2025-11-03T16:15:22.3157552Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-11-03T16:15:22.3157793Z * [new tag] v1.10.2 -> v1.10.2 2025-11-03T16:15:22.3157922Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-11-03T16:15:22.3158015Z * [new tag] v1.11.0 -> v1.11.0 2025-11-03T16:15:22.3158110Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-11-03T16:15:22.3158332Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-11-03T16:15:22.3158441Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-11-03T16:15:22.3159230Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-11-03T16:15:22.3159520Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-11-03T16:15:22.3159998Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-11-03T16:15:22.3160373Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-11-03T16:15:22.3161136Z * [new tag] v1.12.0 -> v1.12.0 2025-11-03T16:15:22.3161315Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-11-03T16:15:22.3163400Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-11-03T16:15:22.3163692Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-11-03T16:15:22.3164253Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-11-03T16:15:22.3164377Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-11-03T16:15:22.3164497Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-11-03T16:15:22.3164728Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-11-03T16:15:22.3165028Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-11-03T16:15:22.3165525Z * [new tag] v1.12.1 -> v1.12.1 2025-11-03T16:15:22.3168708Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-11-03T16:15:22.3168996Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-11-03T16:15:22.3169113Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-11-03T16:15:22.3169212Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-11-03T16:15:22.3169306Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-11-03T16:15:22.3169534Z * [new tag] v1.13.0 -> v1.13.0 2025-11-03T16:15:22.3169640Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-11-03T16:15:22.3169814Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-11-03T16:15:22.3170371Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-11-03T16:15:22.3170928Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-11-03T16:15:22.3171345Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-11-03T16:15:22.3171752Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-11-03T16:15:22.3173558Z * [new tag] v1.13.1 -> v1.13.1 2025-11-03T16:15:22.3173704Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-11-03T16:15:22.3173804Z * [new tag] v1.2.0 -> v1.2.0 2025-11-03T16:15:22.3173951Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-11-03T16:15:22.3174409Z * [new tag] v1.3.0 -> v1.3.0 2025-11-03T16:15:22.3175773Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-11-03T16:15:22.3175913Z * [new tag] v1.3.1 -> v1.3.1 2025-11-03T16:15:22.3176030Z * [new tag] v1.4.0 -> v1.4.0 2025-11-03T16:15:22.3176462Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-11-03T16:15:22.3176861Z * [new tag] v1.4.1 -> v1.4.1 2025-11-03T16:15:22.3178707Z * [new tag] v1.5.0 -> v1.5.0 2025-11-03T16:15:22.3179007Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-11-03T16:15:22.3179120Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-11-03T16:15:22.3179869Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-11-03T16:15:22.3180260Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-11-03T16:15:22.3180788Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-11-03T16:15:22.3181535Z * [new tag] v1.5.1 -> v1.5.1 2025-11-03T16:15:22.3181656Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-11-03T16:15:22.3182096Z * [new tag] v1.6.0 -> v1.6.0 2025-11-03T16:15:22.3184171Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-11-03T16:15:22.3184448Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-11-03T16:15:22.3184569Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-11-03T16:15:22.3184685Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-11-03T16:15:22.3185001Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-11-03T16:15:22.3186086Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-11-03T16:15:22.3186226Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-11-03T16:15:22.3186542Z * [new tag] v1.7.0 -> v1.7.0 2025-11-03T16:15:22.3187801Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-11-03T16:15:22.3187934Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-11-03T16:15:22.3188531Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-11-03T16:15:22.3188861Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-11-03T16:15:22.3189528Z * [new tag] v1.7.1 -> v1.7.1 2025-11-03T16:15:22.3189972Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-11-03T16:15:22.3192492Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-11-03T16:15:22.3192618Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-11-03T16:15:22.3192716Z * [new tag] v1.8.0 -> v1.8.0 2025-11-03T16:15:22.3192832Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-11-03T16:15:22.3192921Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-11-03T16:15:22.3193196Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-11-03T16:15:22.3193652Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-11-03T16:15:22.3194139Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-11-03T16:15:22.3194497Z * [new tag] v1.8.1 -> v1.8.1 2025-11-03T16:15:22.3195501Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-11-03T16:15:22.3195615Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-11-03T16:15:22.3195888Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-11-03T16:15:22.3199352Z * [new tag] v1.8.2 -> v1.8.2 2025-11-03T16:15:22.3199501Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-11-03T16:15:22.3199599Z * [new tag] v1.9.0 -> v1.9.0 2025-11-03T16:15:22.3199702Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-11-03T16:15:22.3199802Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-11-03T16:15:22.3199890Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-11-03T16:15:22.3200151Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-11-03T16:15:22.3200254Z * [new tag] v1.9.1 -> v1.9.1 2025-11-03T16:15:22.3201580Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-11-03T16:15:22.3201857Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-11-03T16:15:22.3201985Z * [new tag] v2.0.0 -> v2.0.0 2025-11-03T16:15:22.3204435Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-11-03T16:15:22.3204723Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-11-03T16:15:22.3204841Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-11-03T16:15:22.3205014Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-11-03T16:15:22.3205319Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-11-03T16:15:22.3205409Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-11-03T16:15:22.3205690Z * [new tag] v2.0.1 -> v2.0.1 2025-11-03T16:15:22.3206646Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-11-03T16:15:22.3206753Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-11-03T16:15:22.3209493Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-11-03T16:15:22.3209619Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-11-03T16:15:22.3209730Z * [new tag] v2.1.0 -> v2.1.0 2025-11-03T16:15:22.3209835Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-11-03T16:15:22.3209935Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-11-03T16:15:22.3210344Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-11-03T16:15:22.3211074Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-11-03T16:15:22.3211432Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-11-03T16:15:22.3211815Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-11-03T16:15:22.3214570Z * [new tag] v2.1.1 -> v2.1.1 2025-11-03T16:15:22.3214854Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-11-03T16:15:22.3215064Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-11-03T16:15:22.3219045Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-11-03T16:15:22.3219329Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-11-03T16:15:22.3219439Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-11-03T16:15:22.3219852Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-11-03T16:15:22.3219971Z * [new tag] v2.1.2 -> v2.1.2 2025-11-03T16:15:22.3220147Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-11-03T16:15:22.3220795Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-11-03T16:15:22.3220926Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-11-03T16:15:22.3221032Z * [new tag] v2.2.0 -> v2.2.0 2025-11-03T16:15:22.3221245Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-11-03T16:15:22.3221991Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-11-03T16:15:22.3222155Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-11-03T16:15:22.3225612Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-11-03T16:15:22.3225759Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-11-03T16:15:22.3225861Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-11-03T16:15:22.3225949Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-11-03T16:15:22.3226038Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-11-03T16:15:22.3226144Z * [new tag] v2.2.1 -> v2.2.1 2025-11-03T16:15:22.3226234Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-11-03T16:15:22.3226583Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-11-03T16:15:22.3226865Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-11-03T16:15:22.3227119Z * [new tag] v2.2.2 -> v2.2.2 2025-11-03T16:15:22.3227783Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-11-03T16:15:22.3227901Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-11-03T16:15:22.3228750Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-11-03T16:15:22.3229098Z * [new tag] v2.3.0 -> v2.3.0 2025-11-03T16:15:22.3229561Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-11-03T16:15:22.3230437Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-11-03T16:15:22.3230805Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-11-03T16:15:22.3231239Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-11-03T16:15:22.3231705Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-11-03T16:15:22.3232780Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-11-03T16:15:22.3233065Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-11-03T16:15:22.3233643Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-11-03T16:15:22.3234036Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-11-03T16:15:22.3234838Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-11-03T16:15:22.3235155Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-11-03T16:15:22.3237897Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-11-03T16:15:22.3238031Z * [new tag] v2.3.1 -> v2.3.1 2025-11-03T16:15:22.3238131Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-11-03T16:15:22.3238228Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-11-03T16:15:22.3238348Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-11-03T16:15:22.3238592Z * [new tag] v2.4.0 -> v2.4.0 2025-11-03T16:15:22.3240066Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-11-03T16:15:22.3240538Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-11-03T16:15:22.3240669Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-11-03T16:15:22.3240838Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-11-03T16:15:22.3242225Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-11-03T16:15:22.3242498Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-11-03T16:15:22.3242620Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-11-03T16:15:22.3243084Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-11-03T16:15:22.3244588Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-11-03T16:15:22.3244872Z * [new tag] v2.4.1 -> v2.4.1 2025-11-03T16:15:22.3245008Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-11-03T16:15:22.3245308Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-11-03T16:15:22.3246839Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-11-03T16:15:22.3247116Z * [new tag] v2.5.0 -> v2.5.0 2025-11-03T16:15:22.3247247Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-11-03T16:15:22.3247508Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-11-03T16:15:22.3248949Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-11-03T16:15:22.3249235Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-11-03T16:15:22.3249348Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-11-03T16:15:22.3249800Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-11-03T16:15:22.3250243Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-11-03T16:15:22.3253920Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-11-03T16:15:22.3254197Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-11-03T16:15:22.3254307Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-11-03T16:15:22.3254491Z * [new tag] v2.5.1 -> v2.5.1 2025-11-03T16:15:22.3254601Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-11-03T16:15:22.3254799Z * [new tag] v2.6.0 -> v2.6.0 2025-11-03T16:15:22.3255115Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-11-03T16:15:22.3255789Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-11-03T16:15:22.3255928Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-11-03T16:15:22.3256031Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-11-03T16:15:22.3256803Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-11-03T16:15:22.3257189Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-11-03T16:15:22.3259429Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-11-03T16:15:22.3259701Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-11-03T16:15:22.3259821Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-11-03T16:15:22.3259993Z * [new tag] v2.7.0 -> v2.7.0 2025-11-03T16:15:22.3260223Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-11-03T16:15:22.3261536Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-11-03T16:15:22.3261829Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-11-03T16:15:22.3261953Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-11-03T16:15:22.3263508Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-11-03T16:15:22.3263974Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-11-03T16:15:22.3264093Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-11-03T16:15:22.3264182Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-11-03T16:15:22.3266105Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-11-03T16:15:22.3266238Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-11-03T16:15:22.3266338Z * [new tag] v2.7.1 -> v2.7.1 2025-11-03T16:15:22.3266601Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-11-03T16:15:22.3267036Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-11-03T16:15:22.3268112Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-11-03T16:15:22.3268213Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-11-03T16:15:22.3269193Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-11-03T16:15:22.3269423Z * [new tag] v2.8.0 -> v2.8.0 2025-11-03T16:15:22.3270030Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-11-03T16:15:22.3270440Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-11-03T16:15:22.3271413Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-11-03T16:15:22.3271736Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-11-03T16:15:22.3272529Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-11-03T16:15:22.3272915Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-11-03T16:15:22.3273992Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-11-03T16:15:22.3274103Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-11-03T16:15:22.3275080Z * [new tag] v2.9.0 -> v2.9.0 2025-11-03T16:15:22.3275345Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-11-03T16:15:22.3278038Z * [new tag] v2.9.0-rc10 -> v2.9.0-rc10 2025-11-03T16:15:22.3278173Z * [new tag] v2.9.0-rc11 -> v2.9.0-rc11 2025-11-03T16:15:22.3278286Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-11-03T16:15:22.3278382Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-11-03T16:15:22.3278522Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-11-03T16:15:22.3279636Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-11-03T16:15:22.3280116Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-11-03T16:15:22.3280489Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-11-03T16:15:22.3281428Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-11-03T16:15:22.3281576Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-11-03T16:15:22.3283546Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-11-03T16:15:22.3283851Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-11-03T16:15:22.3284061Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-11-03T16:15:22.3284271Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-11-03T16:15:22.3284653Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-11-03T16:15:22.3285518Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-11-03T16:15:22.3285672Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-11-03T16:15:22.3287639Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-11-03T16:15:22.3288126Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-11-03T16:15:22.3288320Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-11-03T16:15:22.3288457Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-11-03T16:15:22.3288758Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-11-03T16:15:22.3289950Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-11-03T16:15:22.3290243Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-11-03T16:15:22.3290384Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-11-03T16:15:22.3290828Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-11-03T16:15:22.3292841Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-11-03T16:15:22.3293002Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-11-03T16:15:22.3293117Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-11-03T16:15:22.3293243Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-11-03T16:15:22.3293558Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-11-03T16:15:22.3295677Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-11-03T16:15:22.3295983Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-11-03T16:15:22.3296195Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-11-03T16:15:22.3296568Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-11-03T16:15:22.3296827Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-11-03T16:15:22.3297443Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-11-03T16:15:22.3297588Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-11-03T16:15:22.3297762Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-11-03T16:15:22.3298504Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-11-03T16:15:22.3298855Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-11-03T16:15:22.3300888Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-11-03T16:15:22.3301216Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-11-03T16:15:22.3301420Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-11-03T16:15:22.3301562Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-11-03T16:15:22.3301883Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-11-03T16:15:22.3302670Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-11-03T16:15:22.3302913Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-11-03T16:15:22.3306501Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-11-03T16:15:22.3306794Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-11-03T16:15:22.3306998Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-11-03T16:15:22.3307204Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-11-03T16:15:22.3307345Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-11-03T16:15:22.3307460Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-11-03T16:15:22.3307844Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-11-03T16:15:22.3308482Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-11-03T16:15:22.3308632Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-11-03T16:15:22.3308748Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-11-03T16:15:22.3308868Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-11-03T16:15:22.3309459Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-11-03T16:15:22.3309842Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-11-03T16:15:22.3310721Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-11-03T16:15:22.3311037Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-11-03T16:15:22.3311948Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-11-03T16:15:22.3312177Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-11-03T16:15:22.3312824Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-11-03T16:15:22.3313345Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-11-03T16:15:22.3314292Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-11-03T16:15:22.3314653Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-11-03T16:15:22.3317572Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-11-03T16:15:22.3317900Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-11-03T16:15:22.3318113Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-11-03T16:15:22.3318258Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-11-03T16:15:22.3318450Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-11-03T16:15:22.3318586Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-11-03T16:15:22.3319126Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-11-03T16:15:22.3319277Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-11-03T16:15:22.3319593Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-11-03T16:15:22.3320853Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-11-03T16:15:22.3321148Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-11-03T16:15:22.3321295Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-11-03T16:15:22.3321759Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-11-03T16:15:22.3323096Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-11-03T16:15:22.3323389Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-11-03T16:15:22.3323530Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-11-03T16:15:22.3323835Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-11-03T16:15:22.3324465Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-11-03T16:15:22.3324924Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-11-03T16:15:22.3327974Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-11-03T16:15:22.3328275Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-11-03T16:15:22.3328732Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-11-03T16:15:22.3328935Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-11-03T16:15:22.3329149Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-11-03T16:15:22.3329783Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-11-03T16:15:22.3329924Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-11-03T16:15:22.3330035Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-11-03T16:15:22.3330165Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-11-03T16:15:22.3330304Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-11-03T16:15:22.3331550Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-11-03T16:15:22.3331715Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-11-03T16:15:22.3332181Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-11-03T16:15:22.3334450Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-11-03T16:15:22.3334743Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-11-03T16:15:22.3334964Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-11-03T16:15:22.3335095Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-11-03T16:15:22.3335427Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-11-03T16:15:22.3335663Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-11-03T16:15:22.3336097Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-11-03T16:15:22.3336675Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-11-03T16:15:22.3336796Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-11-03T16:15:22.3339085Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-11-03T16:15:22.3339392Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-11-03T16:15:22.3339587Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-11-03T16:15:22.3339726Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-11-03T16:15:22.3339870Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-11-03T16:15:22.3340098Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-11-03T16:15:22.3340836Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-11-03T16:15:22.3340981Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-11-03T16:15:22.3343106Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-11-03T16:15:22.3343405Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-11-03T16:15:22.3343596Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-11-03T16:15:22.3343795Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-11-03T16:15:22.3343929Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-11-03T16:15:22.3344400Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-11-03T16:15:22.3345540Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-11-03T16:15:22.3345830Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-11-03T16:15:22.3346236Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-11-03T16:15:22.3346372Z * [new tag] viable/strict/1760065682 -> viable/strict/1760065682 2025-11-03T16:15:22.3347682Z * [new tag] viable/strict/1760066894 -> viable/strict/1760066894 2025-11-03T16:15:22.3347834Z * [new tag] viable/strict/1760070345 -> viable/strict/1760070345 2025-11-03T16:15:22.3348135Z * [new tag] viable/strict/1760089782 -> viable/strict/1760089782 2025-11-03T16:15:22.3348637Z * [new tag] viable/strict/1760091921 -> viable/strict/1760091921 2025-11-03T16:15:22.3349165Z * [new tag] viable/strict/1760127924 -> viable/strict/1760127924 2025-11-03T16:15:22.3350052Z * [new tag] viable/strict/1760129489 -> viable/strict/1760129489 2025-11-03T16:15:22.3350374Z * [new tag] viable/strict/1760132980 -> viable/strict/1760132980 2025-11-03T16:15:22.3350925Z * [new tag] viable/strict/1760135060 -> viable/strict/1760135060 2025-11-03T16:15:22.3351407Z * [new tag] viable/strict/1760215782 -> viable/strict/1760215782 2025-11-03T16:15:22.3351859Z * [new tag] viable/strict/1760273849 -> viable/strict/1760273849 2025-11-03T16:15:22.3352670Z * [new tag] viable/strict/1760275517 -> viable/strict/1760275517 2025-11-03T16:15:22.3352891Z * [new tag] viable/strict/1760276979 -> viable/strict/1760276979 2025-11-03T16:15:22.3353410Z * [new tag] viable/strict/1760279007 -> viable/strict/1760279007 2025-11-03T16:15:22.3354003Z * [new tag] viable/strict/1760286328 -> viable/strict/1760286328 2025-11-03T16:15:22.3354502Z * [new tag] viable/strict/1760493304 -> viable/strict/1760493304 2025-11-03T16:15:22.3354826Z * [new tag] viable/strict/1760496298 -> viable/strict/1760496298 2025-11-03T16:15:22.3355702Z * [new tag] viable/strict/1760518396 -> viable/strict/1760518396 2025-11-03T16:15:22.3355837Z * [new tag] viable/strict/1760534864 -> viable/strict/1760534864 2025-11-03T16:15:22.3358982Z * [new tag] viable/strict/1760549062 -> viable/strict/1760549062 2025-11-03T16:15:22.3359136Z * [new tag] viable/strict/1760552799 -> viable/strict/1760552799 2025-11-03T16:15:22.3359264Z * [new tag] viable/strict/1760554355 -> viable/strict/1760554355 2025-11-03T16:15:22.3359386Z * [new tag] viable/strict/1760556275 -> viable/strict/1760556275 2025-11-03T16:15:22.3359533Z * [new tag] viable/strict/1760564979 -> viable/strict/1760564979 2025-11-03T16:15:22.3359816Z * [new tag] viable/strict/1760567049 -> viable/strict/1760567049 2025-11-03T16:15:22.3360282Z * [new tag] viable/strict/1760568585 -> viable/strict/1760568585 2025-11-03T16:15:22.3361437Z * [new tag] viable/strict/1760570630 -> viable/strict/1760570630 2025-11-03T16:15:22.3361868Z * [new tag] viable/strict/1760572180 -> viable/strict/1760572180 2025-11-03T16:15:22.3362118Z * [new tag] viable/strict/1760575094 -> viable/strict/1760575094 2025-11-03T16:15:22.3364695Z * [new tag] viable/strict/1760579709 -> viable/strict/1760579709 2025-11-03T16:15:22.3365003Z * [new tag] viable/strict/1760582614 -> viable/strict/1760582614 2025-11-03T16:15:22.3365239Z * [new tag] viable/strict/1760586815 -> viable/strict/1760586815 2025-11-03T16:15:22.3365465Z * [new tag] viable/strict/1760588829 -> viable/strict/1760588829 2025-11-03T16:15:22.3365605Z * [new tag] viable/strict/1760590200 -> viable/strict/1760590200 2025-11-03T16:15:22.3365891Z * [new tag] viable/strict/1760592311 -> viable/strict/1760592311 2025-11-03T16:15:22.3366434Z * [new tag] viable/strict/1760619733 -> viable/strict/1760619733 2025-11-03T16:15:22.3367052Z * [new tag] viable/strict/1760628335 -> viable/strict/1760628335 2025-11-03T16:15:22.3367413Z * [new tag] viable/strict/1760635490 -> viable/strict/1760635490 2025-11-03T16:15:22.3367983Z * [new tag] viable/strict/1760640743 -> viable/strict/1760640743 2025-11-03T16:15:22.3368353Z * [new tag] viable/strict/1760642528 -> viable/strict/1760642528 2025-11-03T16:15:22.3369092Z * [new tag] viable/strict/1760646330 -> viable/strict/1760646330 2025-11-03T16:15:22.3369718Z * [new tag] viable/strict/1760666101 -> viable/strict/1760666101 2025-11-03T16:15:22.3370265Z * [new tag] viable/strict/1760668990 -> viable/strict/1760668990 2025-11-03T16:15:22.3373237Z * [new tag] viable/strict/1760670600 -> viable/strict/1760670600 2025-11-03T16:15:22.3373397Z * [new tag] viable/strict/1760671704 -> viable/strict/1760671704 2025-11-03T16:15:22.3373520Z * [new tag] viable/strict/1760673121 -> viable/strict/1760673121 2025-11-03T16:15:22.3373630Z * [new tag] viable/strict/1760675352 -> viable/strict/1760675352 2025-11-03T16:15:22.3373750Z * [new tag] viable/strict/1760696731 -> viable/strict/1760696731 2025-11-03T16:15:22.3377430Z * [new tag] viable/strict/1760723515 -> viable/strict/1760723515 2025-11-03T16:15:22.3377750Z * [new tag] viable/strict/1760727234 -> viable/strict/1760727234 2025-11-03T16:15:22.3378085Z * [new tag] viable/strict/1760730578 -> viable/strict/1760730578 2025-11-03T16:15:22.3378348Z * [new tag] viable/strict/1760732726 -> viable/strict/1760732726 2025-11-03T16:15:22.3378873Z * [new tag] viable/strict/1760734180 -> viable/strict/1760734180 2025-11-03T16:15:22.3379034Z * [new tag] viable/strict/1760736251 -> viable/strict/1760736251 2025-11-03T16:15:22.3379148Z * [new tag] viable/strict/1760737772 -> viable/strict/1760737772 2025-11-03T16:15:22.3379260Z * [new tag] viable/strict/1760758005 -> viable/strict/1760758005 2025-11-03T16:15:22.3379385Z * [new tag] viable/strict/1760761532 -> viable/strict/1760761532 2025-11-03T16:15:22.3379609Z * [new tag] viable/strict/1760802581 -> viable/strict/1760802581 2025-11-03T16:15:22.3380369Z * [new tag] viable/strict/1760827772 -> viable/strict/1760827772 2025-11-03T16:15:22.3380794Z * [new tag] viable/strict/1760834524 -> viable/strict/1760834524 2025-11-03T16:15:22.3381181Z * [new tag] viable/strict/1760845009 -> viable/strict/1760845009 2025-11-03T16:15:22.3383076Z * [new tag] viable/strict/1760876836 -> viable/strict/1760876836 2025-11-03T16:15:22.3383388Z * [new tag] viable/strict/1760880329 -> viable/strict/1760880329 2025-11-03T16:15:22.3383644Z * [new tag] viable/strict/1760888987 -> viable/strict/1760888987 2025-11-03T16:15:22.3383850Z * [new tag] viable/strict/1760912664 -> viable/strict/1760912664 2025-11-03T16:15:22.3383999Z * [new tag] viable/strict/1760925321 -> viable/strict/1760925321 2025-11-03T16:15:22.3385164Z * [new tag] viable/strict/1760931488 -> viable/strict/1760931488 2025-11-03T16:15:22.3385315Z * [new tag] viable/strict/1760932693 -> viable/strict/1760932693 2025-11-03T16:15:22.3385601Z * [new tag] viable/strict/1761004184 -> viable/strict/1761004184 2025-11-03T16:15:22.3386080Z * [new tag] viable/strict/1761014748 -> viable/strict/1761014748 2025-11-03T16:15:22.3387312Z * [new tag] viable/strict/1761017491 -> viable/strict/1761017491 2025-11-03T16:15:22.3387625Z * [new tag] viable/strict/1761018806 -> viable/strict/1761018806 2025-11-03T16:15:22.3387744Z * [new tag] viable/strict/1761020754 -> viable/strict/1761020754 2025-11-03T16:15:22.3388300Z * [new tag] viable/strict/1761024303 -> viable/strict/1761024303 2025-11-03T16:15:22.3388679Z * [new tag] viable/strict/1761029582 -> viable/strict/1761029582 2025-11-03T16:15:22.3389439Z * [new tag] viable/strict/1761031535 -> viable/strict/1761031535 2025-11-03T16:15:22.3389630Z * [new tag] viable/strict/1761035196 -> viable/strict/1761035196 2025-11-03T16:15:22.3390671Z * [new tag] viable/strict/1761045825 -> viable/strict/1761045825 2025-11-03T16:15:22.3390888Z * [new tag] viable/strict/1761054796 -> viable/strict/1761054796 2025-11-03T16:15:22.3391784Z * [new tag] viable/strict/1761060314 -> viable/strict/1761060314 2025-11-03T16:15:22.3392121Z * [new tag] viable/strict/1761071198 -> viable/strict/1761071198 2025-11-03T16:15:22.3392633Z * [new tag] viable/strict/1761074628 -> viable/strict/1761074628 2025-11-03T16:15:22.3393150Z * [new tag] viable/strict/1761078351 -> viable/strict/1761078351 2025-11-03T16:15:22.3393957Z * [new tag] viable/strict/1761079822 -> viable/strict/1761079822 2025-11-03T16:15:22.3394155Z * [new tag] viable/strict/1761081873 -> viable/strict/1761081873 2025-11-03T16:15:22.3395179Z * [new tag] viable/strict/1761083392 -> viable/strict/1761083392 2025-11-03T16:15:22.3395360Z * [new tag] viable/strict/1761085465 -> viable/strict/1761085465 2025-11-03T16:15:22.3397377Z * [new tag] viable/strict/1761089099 -> viable/strict/1761089099 2025-11-03T16:15:22.3397532Z * [new tag] viable/strict/1761095535 -> viable/strict/1761095535 2025-11-03T16:15:22.3397678Z * [new tag] viable/strict/1761098119 -> viable/strict/1761098119 2025-11-03T16:15:22.3398178Z * [new tag] viable/strict/1761101330 -> viable/strict/1761101330 2025-11-03T16:15:22.3398878Z * [new tag] viable/strict/1761114425 -> viable/strict/1761114425 2025-11-03T16:15:22.3399259Z * [new tag] viable/strict/1761116036 -> viable/strict/1761116036 2025-11-03T16:15:22.3399792Z * [new tag] viable/strict/1761119379 -> viable/strict/1761119379 2025-11-03T16:15:22.3400309Z * [new tag] viable/strict/1761121601 -> viable/strict/1761121601 2025-11-03T16:15:22.3400870Z * [new tag] viable/strict/1761123234 -> viable/strict/1761123234 2025-11-03T16:15:22.3401346Z * [new tag] viable/strict/1761126621 -> viable/strict/1761126621 2025-11-03T16:15:22.3402143Z * [new tag] viable/strict/1761132259 -> viable/strict/1761132259 2025-11-03T16:15:22.3402319Z * [new tag] viable/strict/1761146746 -> viable/strict/1761146746 2025-11-03T16:15:22.3403254Z * [new tag] viable/strict/1761164752 -> viable/strict/1761164752 2025-11-03T16:15:22.3403545Z * [new tag] viable/strict/1761166198 -> viable/strict/1761166198 2025-11-03T16:15:22.3404051Z * [new tag] viable/strict/1761175424 -> viable/strict/1761175424 2025-11-03T16:15:22.3406784Z * [new tag] viable/strict/1761176983 -> viable/strict/1761176983 2025-11-03T16:15:22.3406924Z * [new tag] viable/strict/1761179891 -> viable/strict/1761179891 2025-11-03T16:15:22.3407063Z * [new tag] viable/strict/1761181930 -> viable/strict/1761181930 2025-11-03T16:15:22.3407174Z * [new tag] viable/strict/1761184516 -> viable/strict/1761184516 2025-11-03T16:15:22.3407290Z * [new tag] viable/strict/1761190179 -> viable/strict/1761190179 2025-11-03T16:15:22.3407783Z * [new tag] viable/strict/1761193558 -> viable/strict/1761193558 2025-11-03T16:15:22.3408853Z * [new tag] viable/strict/1761207990 -> viable/strict/1761207990 2025-11-03T16:15:22.3409001Z * [new tag] viable/strict/1761229539 -> viable/strict/1761229539 2025-11-03T16:15:22.3411922Z * [new tag] viable/strict/1761244031 -> viable/strict/1761244031 2025-11-03T16:15:22.3412214Z * [new tag] viable/strict/1761248986 -> viable/strict/1761248986 2025-11-03T16:15:22.3412451Z * [new tag] viable/strict/1761259791 -> viable/strict/1761259791 2025-11-03T16:15:22.3412613Z * [new tag] viable/strict/1761266139 -> viable/strict/1761266139 2025-11-03T16:15:22.3412827Z * [new tag] viable/strict/1761268316 -> viable/strict/1761268316 2025-11-03T16:15:22.3412964Z * [new tag] viable/strict/1761273805 -> viable/strict/1761273805 2025-11-03T16:15:22.3413173Z * [new tag] viable/strict/1761275261 -> viable/strict/1761275261 2025-11-03T16:15:22.3413709Z * [new tag] viable/strict/1761277913 -> viable/strict/1761277913 2025-11-03T16:15:22.3413992Z * [new tag] viable/strict/1761290701 -> viable/strict/1761290701 2025-11-03T16:15:22.3415338Z * [new tag] viable/strict/1761294396 -> viable/strict/1761294396 2025-11-03T16:15:22.3415490Z * [new tag] viable/strict/1761303047 -> viable/strict/1761303047 2025-11-03T16:15:22.3415652Z * [new tag] viable/strict/1761335388 -> viable/strict/1761335388 2025-11-03T16:15:22.3417674Z * [new tag] viable/strict/1761337551 -> viable/strict/1761337551 2025-11-03T16:15:22.3417996Z * [new tag] viable/strict/1761339007 -> viable/strict/1761339007 2025-11-03T16:15:22.3418205Z * [new tag] viable/strict/1761341050 -> viable/strict/1761341050 2025-11-03T16:15:22.3418346Z * [new tag] viable/strict/1761346188 -> viable/strict/1761346188 2025-11-03T16:15:22.3418578Z * [new tag] viable/strict/1761349792 -> viable/strict/1761349792 2025-11-03T16:15:22.3419251Z * [new tag] viable/strict/1761352620 -> viable/strict/1761352620 2025-11-03T16:15:22.3419425Z * [new tag] viable/strict/1761354730 -> viable/strict/1761354730 2025-11-03T16:15:22.3420284Z * [new tag] viable/strict/1761357298 -> viable/strict/1761357298 2025-11-03T16:15:22.3420416Z * [new tag] viable/strict/1761360201 -> viable/strict/1761360201 2025-11-03T16:15:22.3422776Z * [new tag] viable/strict/1761361753 -> viable/strict/1761361753 2025-11-03T16:15:22.3423073Z * [new tag] viable/strict/1761364351 -> viable/strict/1761364351 2025-11-03T16:15:22.3423268Z * [new tag] viable/strict/1761366338 -> viable/strict/1761366338 2025-11-03T16:15:22.3423459Z * [new tag] viable/strict/1761367802 -> viable/strict/1761367802 2025-11-03T16:15:22.3423589Z * [new tag] viable/strict/1761369889 -> viable/strict/1761369889 2025-11-03T16:15:22.3423711Z * [new tag] viable/strict/1761371385 -> viable/strict/1761371385 2025-11-03T16:15:22.3425272Z * [new tag] viable/strict/1761373581 -> viable/strict/1761373581 2025-11-03T16:15:22.3425426Z * [new tag] viable/strict/1761375054 -> viable/strict/1761375054 2025-11-03T16:15:22.3425772Z * [new tag] viable/strict/1761421785 -> viable/strict/1761421785 2025-11-03T16:15:22.3426271Z * [new tag] viable/strict/1761434614 -> viable/strict/1761434614 2025-11-03T16:15:22.3428769Z * [new tag] viable/strict/1761439254 -> viable/strict/1761439254 2025-11-03T16:15:22.3428914Z * [new tag] viable/strict/1761454187 -> viable/strict/1761454187 2025-11-03T16:15:22.3429243Z * [new tag] viable/strict/1761459991 -> viable/strict/1761459991 2025-11-03T16:15:22.3429371Z * [new tag] viable/strict/1761470668 -> viable/strict/1761470668 2025-11-03T16:15:22.3429965Z * [new tag] viable/strict/1761472188 -> viable/strict/1761472188 2025-11-03T16:15:22.3430402Z * [new tag] viable/strict/1761503178 -> viable/strict/1761503178 2025-11-03T16:15:22.3431312Z * [new tag] viable/strict/1761517492 -> viable/strict/1761517492 2025-11-03T16:15:22.3431854Z * [new tag] viable/strict/1761518981 -> viable/strict/1761518981 2025-11-03T16:15:22.3432008Z * [new tag] viable/strict/1761533609 -> viable/strict/1761533609 2025-11-03T16:15:22.3432447Z * [new tag] viable/strict/1761546438 -> viable/strict/1761546438 2025-11-03T16:15:22.3432830Z * [new tag] viable/strict/1761548133 -> viable/strict/1761548133 2025-11-03T16:15:22.3433455Z * [new tag] viable/strict/1761555186 -> viable/strict/1761555186 2025-11-03T16:15:22.3433742Z * [new tag] viable/strict/1761557178 -> viable/strict/1761557178 2025-11-03T16:15:22.3434265Z * [new tag] viable/strict/1761560772 -> viable/strict/1761560772 2025-11-03T16:15:22.3436942Z * [new tag] viable/strict/1761562266 -> viable/strict/1761562266 2025-11-03T16:15:22.3437101Z * [new tag] viable/strict/1761564260 -> viable/strict/1761564260 2025-11-03T16:15:22.3437216Z * [new tag] viable/strict/1761568072 -> viable/strict/1761568072 2025-11-03T16:15:22.3437504Z * [new tag] viable/strict/1761571683 -> viable/strict/1761571683 2025-11-03T16:15:22.3437633Z * [new tag] viable/strict/1761580199 -> viable/strict/1761580199 2025-11-03T16:15:22.3437764Z * [new tag] viable/strict/1761587383 -> viable/strict/1761587383 2025-11-03T16:15:22.3438052Z * [new tag] viable/strict/1761591165 -> viable/strict/1761591165 2025-11-03T16:15:22.3452674Z * [new tag] viable/strict/1761594575 -> viable/strict/1761594575 2025-11-03T16:15:22.3452977Z * [new tag] viable/strict/1761596710 -> viable/strict/1761596710 2025-11-03T16:15:22.3453104Z * [new tag] viable/strict/1761598189 -> viable/strict/1761598189 2025-11-03T16:15:22.3453238Z * [new tag] viable/strict/1761600254 -> viable/strict/1761600254 2025-11-03T16:15:22.3453354Z * [new tag] viable/strict/1761603879 -> viable/strict/1761603879 2025-11-03T16:15:22.3453489Z * [new tag] viable/strict/1761605429 -> viable/strict/1761605429 2025-11-03T16:15:22.3453601Z * [new tag] viable/strict/1761607468 -> viable/strict/1761607468 2025-11-03T16:15:22.3453720Z * [new tag] viable/strict/1761608983 -> viable/strict/1761608983 2025-11-03T16:15:22.3453831Z * [new tag] viable/strict/1761611846 -> viable/strict/1761611846 2025-11-03T16:15:22.3453942Z * [new tag] viable/strict/1761613922 -> viable/strict/1761613922 2025-11-03T16:15:22.3454061Z * [new tag] viable/strict/1761616504 -> viable/strict/1761616504 2025-11-03T16:15:22.3454181Z * [new tag] viable/strict/1761619599 -> viable/strict/1761619599 2025-11-03T16:15:22.3454299Z * [new tag] viable/strict/1761686693 -> viable/strict/1761686693 2025-11-03T16:15:22.3454420Z * [new tag] viable/strict/1761688179 -> viable/strict/1761688179 2025-11-03T16:15:22.3454540Z * [new tag] viable/strict/1761691973 -> viable/strict/1761691973 2025-11-03T16:15:22.3454650Z * [new tag] viable/strict/1761693884 -> viable/strict/1761693884 2025-11-03T16:15:22.3454856Z * [new tag] viable/strict/1761695389 -> viable/strict/1761695389 2025-11-03T16:15:22.3454976Z * [new tag] viable/strict/1761698408 -> viable/strict/1761698408 2025-11-03T16:15:22.3455087Z * [new tag] viable/strict/1761702931 -> viable/strict/1761702931 2025-11-03T16:15:22.3455206Z * [new tag] viable/strict/1761706307 -> viable/strict/1761706307 2025-11-03T16:15:22.3455315Z * [new tag] viable/strict/1761709065 -> viable/strict/1761709065 2025-11-03T16:15:22.3455433Z * [new tag] viable/strict/1761710285 -> viable/strict/1761710285 2025-11-03T16:15:22.3455544Z * [new tag] viable/strict/1761711983 -> viable/strict/1761711983 2025-11-03T16:15:22.3455656Z * [new tag] viable/strict/1761713514 -> viable/strict/1761713514 2025-11-03T16:15:22.3455776Z * [new tag] viable/strict/1761715523 -> viable/strict/1761715523 2025-11-03T16:15:22.3455890Z * [new tag] viable/strict/1761727973 -> viable/strict/1761727973 2025-11-03T16:15:22.3456011Z * [new tag] viable/strict/1761751558 -> viable/strict/1761751558 2025-11-03T16:15:22.3456120Z * [new tag] viable/strict/1761755187 -> viable/strict/1761755187 2025-11-03T16:15:22.3456231Z * [new tag] viable/strict/1761756826 -> viable/strict/1761756826 2025-11-03T16:15:22.3456351Z * [new tag] viable/strict/1761769551 -> viable/strict/1761769551 2025-11-03T16:15:22.3456461Z * [new tag] viable/strict/1761771032 -> viable/strict/1761771032 2025-11-03T16:15:22.3456584Z * [new tag] viable/strict/1761773101 -> viable/strict/1761773101 2025-11-03T16:15:22.3456737Z * [new tag] viable/strict/1761781792 -> viable/strict/1761781792 2025-11-03T16:15:22.3456857Z * [new tag] viable/strict/1761784788 -> viable/strict/1761784788 2025-11-03T16:15:22.3456970Z * [new tag] viable/strict/1761786740 -> viable/strict/1761786740 2025-11-03T16:15:22.3457090Z * [new tag] viable/strict/1761789332 -> viable/strict/1761789332 2025-11-03T16:15:22.3457210Z * [new tag] viable/strict/1761792569 -> viable/strict/1761792569 2025-11-03T16:15:22.3457320Z * [new tag] viable/strict/1761795289 -> viable/strict/1761795289 2025-11-03T16:15:22.3457443Z * [new tag] viable/strict/1761798345 -> viable/strict/1761798345 2025-11-03T16:15:22.3457555Z * [new tag] viable/strict/1761799827 -> viable/strict/1761799827 2025-11-03T16:15:22.3457664Z * [new tag] viable/strict/1761805604 -> viable/strict/1761805604 2025-11-03T16:15:22.3457794Z * [new tag] viable/strict/1761807202 -> viable/strict/1761807202 2025-11-03T16:15:22.3458254Z * [new tag] viable/strict/1761809094 -> viable/strict/1761809094 2025-11-03T16:15:22.3458439Z * [new tag] viable/strict/1761810576 -> viable/strict/1761810576 2025-11-03T16:15:22.3458907Z * [new tag] viable/strict/1761812771 -> viable/strict/1761812771 2025-11-03T16:15:22.3460501Z * [new tag] viable/strict/1761814363 -> viable/strict/1761814363 2025-11-03T16:15:22.3460639Z * [new tag] viable/strict/1761857410 -> viable/strict/1761857410 2025-11-03T16:15:22.3460759Z * [new tag] viable/strict/1761860985 -> viable/strict/1761860985 2025-11-03T16:15:22.3460905Z * [new tag] viable/strict/1761863094 -> viable/strict/1761863094 2025-11-03T16:15:22.3461313Z * [new tag] viable/strict/1761864590 -> viable/strict/1761864590 2025-11-03T16:15:22.3462851Z * [new tag] viable/strict/1761866675 -> viable/strict/1761866675 2025-11-03T16:15:22.3463156Z * [new tag] viable/strict/1761868178 -> viable/strict/1761868178 2025-11-03T16:15:22.3463566Z * [new tag] viable/strict/1761871111 -> viable/strict/1761871111 2025-11-03T16:15:22.3463706Z * [new tag] viable/strict/1761873126 -> viable/strict/1761873126 2025-11-03T16:15:22.3463827Z * [new tag] viable/strict/1761875714 -> viable/strict/1761875714 2025-11-03T16:15:22.3464330Z * [new tag] viable/strict/1761878924 -> viable/strict/1761878924 2025-11-03T16:15:22.3464979Z * [new tag] viable/strict/1761881727 -> viable/strict/1761881727 2025-11-03T16:15:22.3465497Z * [new tag] viable/strict/1761882959 -> viable/strict/1761882959 2025-11-03T16:15:22.3465721Z * [new tag] viable/strict/1761886268 -> viable/strict/1761886268 2025-11-03T16:15:22.3466383Z * [new tag] viable/strict/1761893641 -> viable/strict/1761893641 2025-11-03T16:15:22.3466692Z * [new tag] viable/strict/1761931517 -> viable/strict/1761931517 2025-11-03T16:15:22.3467154Z * [new tag] viable/strict/1761933080 -> viable/strict/1761933080 2025-11-03T16:15:22.3469174Z * [new tag] viable/strict/1761935217 -> viable/strict/1761935217 2025-11-03T16:15:22.3469468Z * [new tag] viable/strict/1761938533 -> viable/strict/1761938533 2025-11-03T16:15:22.3469706Z * [new tag] viable/strict/1761940184 -> viable/strict/1761940184 2025-11-03T16:15:22.3469843Z * [new tag] viable/strict/1761942338 -> viable/strict/1761942338 2025-11-03T16:15:22.3470041Z * [new tag] viable/strict/1761946100 -> viable/strict/1761946100 2025-11-03T16:15:22.3470177Z * [new tag] viable/strict/1761947374 -> viable/strict/1761947374 2025-11-03T16:15:22.3470715Z * [new tag] viable/strict/1761950978 -> viable/strict/1761950978 2025-11-03T16:15:22.3471258Z * [new tag] viable/strict/1761957727 -> viable/strict/1761957727 2025-11-03T16:15:22.3471559Z * [new tag] viable/strict/1761959532 -> viable/strict/1761959532 2025-11-03T16:15:22.3472373Z * [new tag] viable/strict/1761965366 -> viable/strict/1761965366 2025-11-03T16:15:22.3472504Z * [new tag] viable/strict/1761968066 -> viable/strict/1761968066 2025-11-03T16:15:22.3475117Z * [new tag] viable/strict/1761969322 -> viable/strict/1761969322 2025-11-03T16:15:22.3475416Z * [new tag] viable/strict/1761974723 -> viable/strict/1761974723 2025-11-03T16:15:22.3475682Z * [new tag] viable/strict/1761981837 -> viable/strict/1761981837 2025-11-03T16:15:22.3475827Z * [new tag] viable/strict/1761985546 -> viable/strict/1761985546 2025-11-03T16:15:22.3476049Z * [new tag] viable/strict/1761987030 -> viable/strict/1761987030 2025-11-03T16:15:22.3476185Z * [new tag] viable/strict/1762003554 -> viable/strict/1762003554 2025-11-03T16:15:22.3476432Z * [new tag] viable/strict/1762021560 -> viable/strict/1762021560 2025-11-03T16:15:22.3476859Z * [new tag] viable/strict/1762032190 -> viable/strict/1762032190 2025-11-03T16:15:22.3477464Z * [new tag] viable/strict/1762040981 -> viable/strict/1762040981 2025-11-03T16:15:22.3477858Z * [new tag] viable/strict/1762048525 -> viable/strict/1762048525 2025-11-03T16:15:22.3480555Z * [new tag] viable/strict/1762104223 -> viable/strict/1762104223 2025-11-03T16:15:22.3480872Z * [new tag] viable/strict/1762105778 -> viable/strict/1762105778 2025-11-03T16:15:22.3481112Z * [new tag] viable/strict/1762115109 -> viable/strict/1762115109 2025-11-03T16:15:22.3481277Z * [new tag] viable/strict/1762125840 -> viable/strict/1762125840 2025-11-03T16:15:22.3481395Z * [new tag] viable/strict/1762127377 -> viable/strict/1762127377 2025-11-03T16:15:22.3481826Z * [new tag] viable/strict/1762134925 -> viable/strict/1762134925 2025-11-03T16:15:22.3482427Z * [new tag] viable/strict/1762138338 -> viable/strict/1762138338 2025-11-03T16:15:22.3482584Z * [new tag] viable/strict/1762148993 -> viable/strict/1762148993 2025-11-03T16:15:22.3482707Z * [new tag] viable/strict/1762152871 -> viable/strict/1762152871 2025-11-03T16:15:22.3483029Z * [new tag] viable/strict/1762156183 -> viable/strict/1762156183 2025-11-03T16:15:22.3483851Z * [new tag] viable/strict/1762163457 -> viable/strict/1762163457 2025-11-03T16:15:22.3483997Z * [new tag] viable/strict/1762165569 -> viable/strict/1762165569 2025-11-03T16:15:22.3486999Z * [new tag] viable/strict/1762169035 -> viable/strict/1762169035 2025-11-03T16:15:22.3487310Z * [new tag] viable/strict/1762174936 -> viable/strict/1762174936 2025-11-03T16:15:22.3487509Z * [new tag] whc_flight_1 -> whc_flight_1 2025-11-03T16:15:22.3487629Z * [new tag] whc_flight_2 -> whc_flight_2 2025-11-03T16:15:22.3487804Z * [new tag] whc_flight_4 -> whc_flight_4 2025-11-03T16:15:22.3962357Z [command]/usr/bin/git rev-parse --verify --quiet 3f6538febd727b782e6e13cfd026a309fb14351d^{object} 2025-11-03T16:15:22.3990693Z 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:15:22.3991968Z ##[endgroup] 2025-11-03T16:15:22.3992758Z ##[group]Determining the checkout info 2025-11-03T16:15:22.3992976Z ##[endgroup] 2025-11-03T16:15:22.3993504Z [command]/usr/bin/git sparse-checkout disable 2025-11-03T16:15:22.4032096Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-11-03T16:15:22.4063748Z ##[group]Checking out the ref 2025-11-03T16:15:22.4068063Z [command]/usr/bin/git checkout --progress --force 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:15:23.4436576Z Note: switching to '3f6538febd727b782e6e13cfd026a309fb14351d'. 2025-11-03T16:15:23.4437385Z 2025-11-03T16:15:23.4437567Z You are in 'detached HEAD' state. You can look around, make experimental 2025-11-03T16:15:23.4437947Z changes and commit them, and you can discard any commits you make in this 2025-11-03T16:15:23.4438353Z state without impacting any branches by switching back to a branch. 2025-11-03T16:15:23.4438576Z 2025-11-03T16:15:23.4438733Z If you want to create a new branch to retain commits you create, you may 2025-11-03T16:15:23.4439040Z do so (now or later) by using -c with the switch command. Example: 2025-11-03T16:15:23.4439205Z 2025-11-03T16:15:23.4439294Z git switch -c 2025-11-03T16:15:23.4439419Z 2025-11-03T16:15:23.4439493Z Or undo this operation with: 2025-11-03T16:15:23.4439645Z 2025-11-03T16:15:23.4439710Z git switch - 2025-11-03T16:15:23.4439805Z 2025-11-03T16:15:23.4439950Z Turn off this advice by setting config variable advice.detachedHead to false 2025-11-03T16:15:23.4440156Z 2025-11-03T16:15:23.4444572Z HEAD is now at 3f6538febd7 Remove tools from BC linter (#166858) 2025-11-03T16:15:23.4503288Z ##[endgroup] 2025-11-03T16:15:23.4508825Z ##[group]Setting up auth for fetching submodules 2025-11-03T16:15:23.4510887Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-11-03T16:15:23.4582140Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-11-03T16:15:23.4611242Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-11-03T16:15:23.4640977Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-11-03T16:15:23.4667962Z ##[endgroup] 2025-11-03T16:15:23.4672419Z ##[group]Fetching submodules 2025-11-03T16:15:23.4676496Z [command]/usr/bin/git submodule sync --recursive 2025-11-03T16:15:23.4991384Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-11-03T16:15:23.5307793Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-11-03T16:15:23.5308608Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-11-03T16:15:23.5309170Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-11-03T16:15:23.5309703Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-11-03T16:15:23.5310201Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-11-03T16:15:23.5310875Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-11-03T16:15:23.5311546Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-11-03T16:15:23.5312055Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-11-03T16:15:23.5313002Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-11-03T16:15:23.5321852Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-11-03T16:15:23.5323696Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-11-03T16:15:23.5324382Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-11-03T16:15:23.5329852Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-11-03T16:15:23.5331750Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-11-03T16:15:23.5337436Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-11-03T16:15:23.5342657Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-11-03T16:15:23.5345202Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-11-03T16:15:23.5345858Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-11-03T16:15:23.5348955Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:15:23.5349543Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-11-03T16:15:23.5350061Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-11-03T16:15:23.5350565Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-11-03T16:15:23.5351043Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-11-03T16:15:23.5351519Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-11-03T16:15:23.5352027Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-11-03T16:15:23.5352563Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-11-03T16:15:23.5353074Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-11-03T16:15:23.5353550Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-11-03T16:15:23.5354660Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-11-03T16:15:23.5358566Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-11-03T16:15:23.5359215Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-11-03T16:15:23.5363686Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-11-03T16:15:23.5364505Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-11-03T16:15:23.5369885Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-11-03T16:15:23.5371184Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-11-03T16:15:23.5376374Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-11-03T16:15:23.5377111Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-11-03T16:15:23.5407134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-11-03T16:15:23.7887463Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-11-03T16:15:23.7887983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-11-03T16:15:23.7888676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-11-03T16:15:23.7912150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-11-03T16:15:24.0834700Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-11-03T16:15:24.0835555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-11-03T16:15:24.0836317Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-11-03T16:15:24.0837093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-11-03T16:15:24.0837866Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-11-03T16:15:24.0838617Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-11-03T16:15:24.0839446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-11-03T16:15:24.0840218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-11-03T16:15:24.0841027Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-11-03T16:15:24.1045436Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-11-03T16:15:24.2553310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-11-03T16:15:24.2554209Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-11-03T16:15:24.2885916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-11-03T16:15:25.3554390Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-11-03T16:15:25.3555026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-11-03T16:15:25.3555517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-11-03T16:15:25.3555989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-11-03T16:15:25.3556645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-11-03T16:15:25.3557082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-11-03T16:15:25.3557548Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-11-03T16:15:25.3558006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-11-03T16:15:25.3558441Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-11-03T16:15:25.4555961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-11-03T16:15:37.2137339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-11-03T16:15:37.2142943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-11-03T16:15:37.2144832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-11-03T16:15:37.2145375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-11-03T16:15:37.2150851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-11-03T16:15:37.2151472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-11-03T16:15:37.2152419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-11-03T16:15:37.2156465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-11-03T16:15:37.2159167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-11-03T16:15:37.2273625Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-11-03T16:15:37.2383410Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-11-03T16:15:37.2475604Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-11-03T16:15:37.2672800Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-11-03T16:15:37.3305419Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-11-03T16:15:37.3757164Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-11-03T16:15:37.8924487Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-11-03T16:15:38.0294490Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-11-03T16:15:38.0311879Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:15:38.0340196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-11-03T16:15:41.9796283Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-11-03T16:15:41.9989007Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-11-03T16:15:42.2527431Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-11-03T16:15:42.2920188Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-11-03T16:15:42.3719608Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-11-03T16:15:42.4129566Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396' 2025-11-03T16:15:42.9189867Z Submodule path 'third_party/cutlass': checked out 'f3fde58372d33e9a5650ba7b80fc48b3b49d40c8' 2025-11-03T16:15:43.0341917Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4' 2025-11-03T16:15:43.0359804Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-11-03T16:15:43.0360430Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:15:43.0361200Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:15:43.0363050Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-11-03T16:15:43.0363632Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-11-03T16:15:43.0364483Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:15:43.0365925Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-11-03T16:15:43.0395772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-11-03T16:15:44.3936974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-11-03T16:15:44.3938227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-11-03T16:15:44.3938714Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-11-03T16:15:44.4936824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-11-03T16:15:46.7746598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-11-03T16:15:46.8750654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-11-03T16:15:49.7511113Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-11-03T16:15:50.0070342Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-11-03T16:15:50.0906888Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-11-03T16:15:50.5788897Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8' 2025-11-03T16:15:50.6176543Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-11-03T16:15:50.6279038Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-11-03T16:15:50.7126999Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-11-03T16:15:50.7678257Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-11-03T16:15:50.7694834Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:15:50.7699422Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:15:50.7725827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-11-03T16:15:54.5940748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-11-03T16:15:54.7697847Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-11-03T16:15:55.2116143Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-11-03T16:15:55.3156955Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-11-03T16:15:55.3417672Z Submodule path 'third_party/fmt': checked out 'e424e3f2e607da02742f73db84873b8084fc714c' 2025-11-03T16:15:55.3745678Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-11-03T16:15:55.3949277Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-11-03T16:15:55.4325997Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-11-03T16:15:55.4443527Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-11-03T16:15:55.4457028Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-11-03T16:15:55.4484168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-11-03T16:16:12.1286610Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-11-03T16:16:12.1464336Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-11-03T16:16:12.2267054Z Submodule path 'third_party/kineto': checked out '6fcbc53d33dd275c0aba1e5d7701d471b7f6eeb3' 2025-11-03T16:16:12.2283445Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:12.2285816Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:12.2286631Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:12.2313401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-11-03T16:16:13.6309522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-11-03T16:16:13.7777571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-11-03T16:16:13.8512050Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-11-03T16:16:13.8528129Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:13.8529749Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:13.8530627Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:13.8535928Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:13.8536829Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:13.8537610Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:13.8542519Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:13.8547979Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:13.8549402Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:13.8570789Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-11-03T16:16:16.0284214Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-11-03T16:16:16.0285140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-11-03T16:16:16.0285919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-11-03T16:16:16.0286561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-11-03T16:16:16.0287210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-11-03T16:16:16.0287827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-11-03T16:16:16.0288452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-11-03T16:16:16.1287476Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-11-03T16:16:20.7281785Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-11-03T16:16:20.7440416Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-11-03T16:16:20.7745895Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-11-03T16:16:20.7869280Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-11-03T16:16:20.7882322Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:20.7910979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-11-03T16:16:21.0775566Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-11-03T16:16:21.0935957Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-11-03T16:16:21.1314101Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-11-03T16:16:21.2106723Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-11-03T16:16:21.2253755Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-11-03T16:16:21.2391107Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-11-03T16:16:21.2408850Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:21.2409849Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:21.2439816Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-11-03T16:16:23.3610517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-11-03T16:16:23.5692815Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-11-03T16:16:23.6069688Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-11-03T16:16:23.6344803Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-11-03T16:16:23.6724268Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-11-03T16:16:23.7164317Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe' 2025-11-03T16:16:23.7483509Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-11-03T16:16:23.8391885Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-11-03T16:16:24.1114625Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-11-03T16:16:24.1145958Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:24.1170691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-11-03T16:16:25.2370055Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-11-03T16:16:25.2885901Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-11-03T16:16:25.2900873Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:25.2901777Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:25.2902478Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:25.2903208Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:25.2904036Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:25.2904941Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:25.2906494Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:25.2907210Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:25.2942527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-11-03T16:16:25.7408624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-11-03T16:16:25.7409772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-11-03T16:16:25.7411062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-11-03T16:16:25.7412079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-11-03T16:16:25.8409829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-11-03T16:16:26.3379793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-11-03T16:16:32.0902190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-11-03T16:16:32.8308427Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-11-03T16:16:32.8645651Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-11-03T16:16:32.8799247Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-11-03T16:16:32.9664048Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-11-03T16:16:32.9784205Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-11-03T16:16:32.9913080Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-11-03T16:16:33.0044896Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-11-03T16:16:33.0057748Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:33.0062997Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:33.0089033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-11-03T16:16:34.9838191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-11-03T16:16:35.2081403Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-11-03T16:16:35.2466130Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-11-03T16:16:35.5584491Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-11-03T16:16:35.5690817Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-11-03T16:16:35.7789379Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-11-03T16:16:35.7808378Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:35.7813782Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:35.7840772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-11-03T16:16:36.2906315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-11-03T16:16:36.7106005Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-11-03T16:16:36.7707523Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-11-03T16:16:36.7791119Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-11-03T16:16:36.7897510Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-11-03T16:16:36.8230353Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-11-03T16:16:36.8468680Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-11-03T16:16:36.8831823Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-11-03T16:16:36.9045509Z Submodule path 'third_party/tensorpipe': checked out 'af0118d13e52f5a08841464a768e01a0bf3e3075' 2025-11-03T16:16:36.9057237Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:36.9058081Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:36.9063443Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:36.9066047Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:36.9092225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-11-03T16:16:37.7847610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-11-03T16:16:37.8848714Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-11-03T16:16:38.0609458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-11-03T16:16:38.1084854Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-11-03T16:16:38.1214822Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-11-03T16:16:38.1816599Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-11-03T16:16:38.2057511Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-11-03T16:16:38.2069547Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:38.2097293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-11-03T16:16:38.4283719Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-11-03T16:16:38.4328179Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-11-03T16:16:38.4642014Z Entering 'android/libs/fbjni' 2025-11-03T16:16:38.4681476Z Entering 'third_party/FP16' 2025-11-03T16:16:38.4721399Z Entering 'third_party/FXdiv' 2025-11-03T16:16:38.4761405Z Entering 'third_party/NNPACK' 2025-11-03T16:16:38.4798666Z Entering 'third_party/NVTX' 2025-11-03T16:16:38.4841511Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:16:38.4880290Z Entering 'third_party/XNNPACK' 2025-11-03T16:16:38.4931104Z Entering 'third_party/aiter' 2025-11-03T16:16:38.4968732Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:16:38.5013557Z Entering 'third_party/benchmark' 2025-11-03T16:16:38.5055262Z Entering 'third_party/composable_kernel' 2025-11-03T16:16:38.5103529Z Entering 'third_party/cpp-httplib' 2025-11-03T16:16:38.5152972Z Entering 'third_party/cpuinfo' 2025-11-03T16:16:38.5192061Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:16:38.5230522Z Entering 'third_party/cutlass' 2025-11-03T16:16:38.5276959Z Entering 'third_party/fbgemm' 2025-11-03T16:16:38.5328341Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:16:38.5367604Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:16:38.5414789Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:16:38.5452687Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:16:38.5499746Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:16:38.5539071Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:16:38.5578234Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:16:38.5622098Z Entering 'third_party/flash-attention' 2025-11-03T16:16:38.5660475Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:16:38.5704588Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:16:38.5751257Z Entering 'third_party/flatbuffers' 2025-11-03T16:16:38.5795578Z Entering 'third_party/fmt' 2025-11-03T16:16:38.5833494Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:16:38.5872154Z Entering 'third_party/gloo' 2025-11-03T16:16:38.5907177Z Entering 'third_party/googletest' 2025-11-03T16:16:38.5950506Z Entering 'third_party/ideep' 2025-11-03T16:16:38.5990678Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:16:38.6037167Z Entering 'third_party/ittapi' 2025-11-03T16:16:38.6080107Z Entering 'third_party/kineto' 2025-11-03T16:16:38.6115423Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:38.6158281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:38.6197859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:38.6238513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:38.6273389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:38.6311175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:38.6353580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:38.6393183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:38.6441027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:38.6481400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:38.6520053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:38.6557474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:38.6600321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:38.6644896Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:38.6683813Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:38.6732009Z Entering 'third_party/kleidiai' 2025-11-03T16:16:38.6771584Z Entering 'third_party/mimalloc' 2025-11-03T16:16:38.6811831Z Entering 'third_party/nlohmann' 2025-11-03T16:16:38.6855252Z Entering 'third_party/onnx' 2025-11-03T16:16:38.6906504Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:38.6950805Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:16:38.6991410Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:38.7029944Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:38.7068855Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:38.7107655Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:38.7149668Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:38.7190588Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:38.7227078Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:38.7265307Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:38.7302567Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:38.7346147Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:38.7400315Z Entering 'third_party/pocketfft' 2025-11-03T16:16:38.7444494Z Entering 'third_party/protobuf' 2025-11-03T16:16:38.7485265Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:38.7525272Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:38.7567530Z Entering 'third_party/psimd' 2025-11-03T16:16:38.7604764Z Entering 'third_party/pthreadpool' 2025-11-03T16:16:38.7648370Z Entering 'third_party/pybind11' 2025-11-03T16:16:38.7691663Z Entering 'third_party/python-peachpy' 2025-11-03T16:16:38.7730455Z Entering 'third_party/sleef' 2025-11-03T16:16:38.7770210Z Entering 'third_party/tensorpipe' 2025-11-03T16:16:38.7808785Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:38.7849280Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:38.7887966Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:38.7928023Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:38.7964273Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:38.8030498Z ##[endgroup] 2025-11-03T16:16:38.8031001Z ##[group]Persisting credentials for submodules 2025-11-03T16:16:38.8031906Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-11-03T16:16:38.8339872Z Entering 'android/libs/fbjni' 2025-11-03T16:16:38.8393007Z Entering 'third_party/FP16' 2025-11-03T16:16:38.8453032Z Entering 'third_party/FXdiv' 2025-11-03T16:16:38.8505158Z Entering 'third_party/NNPACK' 2025-11-03T16:16:38.8558536Z Entering 'third_party/NVTX' 2025-11-03T16:16:38.8617546Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:16:38.8670055Z Entering 'third_party/XNNPACK' 2025-11-03T16:16:38.8735542Z Entering 'third_party/aiter' 2025-11-03T16:16:38.8788279Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:16:38.8848235Z Entering 'third_party/benchmark' 2025-11-03T16:16:38.8899640Z Entering 'third_party/composable_kernel' 2025-11-03T16:16:38.8960687Z Entering 'third_party/cpp-httplib' 2025-11-03T16:16:38.9013476Z Entering 'third_party/cpuinfo' 2025-11-03T16:16:38.9068024Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:16:38.9125288Z Entering 'third_party/cutlass' 2025-11-03T16:16:38.9190364Z Entering 'third_party/fbgemm' 2025-11-03T16:16:38.9242427Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:16:38.9295067Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:16:38.9356620Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:16:38.9408677Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:16:38.9469814Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:16:38.9529462Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:16:38.9583190Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:16:38.9643901Z Entering 'third_party/flash-attention' 2025-11-03T16:16:38.9693727Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:16:38.9749410Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:16:38.9815239Z Entering 'third_party/flatbuffers' 2025-11-03T16:16:38.9869697Z Entering 'third_party/fmt' 2025-11-03T16:16:38.9925055Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:16:38.9982458Z Entering 'third_party/gloo' 2025-11-03T16:16:39.0038612Z Entering 'third_party/googletest' 2025-11-03T16:16:39.0092409Z Entering 'third_party/ideep' 2025-11-03T16:16:39.0147574Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:16:39.0210026Z Entering 'third_party/ittapi' 2025-11-03T16:16:39.0265210Z Entering 'third_party/kineto' 2025-11-03T16:16:39.0316806Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:39.0369004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:39.0425400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:39.0479864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:39.0532006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:39.0582962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:39.0636228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:39.0688923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:39.0746217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:39.0802411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:39.0855414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:39.0907641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:39.0962756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:39.1022327Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:39.1075398Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:39.1132714Z Entering 'third_party/kleidiai' 2025-11-03T16:16:39.1187750Z Entering 'third_party/mimalloc' 2025-11-03T16:16:39.1242924Z Entering 'third_party/nlohmann' 2025-11-03T16:16:39.1297835Z Entering 'third_party/onnx' 2025-11-03T16:16:39.1364871Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:39.1423199Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:16:39.1477943Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:39.1531393Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:39.1586560Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:39.1639772Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:39.1695125Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:39.1748040Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:39.1802735Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:39.1853780Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:39.1909164Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:39.1969733Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:39.2038284Z Entering 'third_party/pocketfft' 2025-11-03T16:16:39.2093686Z Entering 'third_party/protobuf' 2025-11-03T16:16:39.2145981Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:39.2200478Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:39.2256898Z Entering 'third_party/psimd' 2025-11-03T16:16:39.2310308Z Entering 'third_party/pthreadpool' 2025-11-03T16:16:39.2365636Z Entering 'third_party/pybind11' 2025-11-03T16:16:39.2422017Z Entering 'third_party/python-peachpy' 2025-11-03T16:16:39.2470753Z Entering 'third_party/sleef' 2025-11-03T16:16:39.2526323Z Entering 'third_party/tensorpipe' 2025-11-03T16:16:39.2577860Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:39.2631307Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:39.2684370Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:39.2739410Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:39.2796759Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:39.2867912Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-11-03T16:16:39.3188341Z Entering 'android/libs/fbjni' 2025-11-03T16:16:39.3235141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-11-03T16:16:39.3253979Z Entering 'third_party/FP16' 2025-11-03T16:16:39.3302572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-11-03T16:16:39.3325941Z Entering 'third_party/FXdiv' 2025-11-03T16:16:39.3372488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-11-03T16:16:39.3391627Z Entering 'third_party/NNPACK' 2025-11-03T16:16:39.3436861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-11-03T16:16:39.3455760Z Entering 'third_party/NVTX' 2025-11-03T16:16:39.3503413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-11-03T16:16:39.3522972Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:16:39.3569463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-11-03T16:16:39.3585611Z Entering 'third_party/XNNPACK' 2025-11-03T16:16:39.3636840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-11-03T16:16:39.3663482Z Entering 'third_party/aiter' 2025-11-03T16:16:39.3710987Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-11-03T16:16:39.3728723Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:16:39.3776003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-11-03T16:16:39.3799546Z Entering 'third_party/benchmark' 2025-11-03T16:16:39.3851201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-11-03T16:16:39.3865769Z Entering 'third_party/composable_kernel' 2025-11-03T16:16:39.3910221Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-11-03T16:16:39.3936261Z Entering 'third_party/cpp-httplib' 2025-11-03T16:16:39.3979362Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-11-03T16:16:39.3997864Z Entering 'third_party/cpuinfo' 2025-11-03T16:16:39.4044001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-11-03T16:16:39.4061575Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:16:39.4110898Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-11-03T16:16:39.4131836Z Entering 'third_party/cutlass' 2025-11-03T16:16:39.4179921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-11-03T16:16:39.4205972Z Entering 'third_party/fbgemm' 2025-11-03T16:16:39.4258877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-11-03T16:16:39.4276044Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:16:39.4324029Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-11-03T16:16:39.4342539Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:16:39.4386869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-11-03T16:16:39.4409506Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:16:39.4454926Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-11-03T16:16:39.4472403Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:16:39.4520762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-11-03T16:16:39.4546172Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:16:39.4595136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-11-03T16:16:39.4610861Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:16:39.4658862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-11-03T16:16:39.4675515Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:16:39.4722424Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-11-03T16:16:39.4746618Z Entering 'third_party/flash-attention' 2025-11-03T16:16:39.4789893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-11-03T16:16:39.4807582Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:16:39.4851662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-11-03T16:16:39.4873936Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:16:39.4922005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-11-03T16:16:39.4950559Z Entering 'third_party/flatbuffers' 2025-11-03T16:16:39.4998614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-11-03T16:16:39.5020487Z Entering 'third_party/fmt' 2025-11-03T16:16:39.5061530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-11-03T16:16:39.5079172Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:16:39.5127368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-11-03T16:16:39.5145968Z Entering 'third_party/gloo' 2025-11-03T16:16:39.5190860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-11-03T16:16:39.5210780Z Entering 'third_party/googletest' 2025-11-03T16:16:39.5257881Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-11-03T16:16:39.5276261Z Entering 'third_party/ideep' 2025-11-03T16:16:39.5324793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-11-03T16:16:39.5342603Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:16:39.5386039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-11-03T16:16:39.5408402Z Entering 'third_party/ittapi' 2025-11-03T16:16:39.5457039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-11-03T16:16:39.5474914Z Entering 'third_party/kineto' 2025-11-03T16:16:39.5522053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-11-03T16:16:39.5540321Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:39.5586890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-11-03T16:16:39.5601928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:39.5648031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-11-03T16:16:39.5668293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:39.5716160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-11-03T16:16:39.5735323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:39.5780710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-11-03T16:16:39.5800999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:39.5849187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-11-03T16:16:39.5864575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:39.5911626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-11-03T16:16:39.5934333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:39.5980126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-11-03T16:16:39.5998411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:39.6042758Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-11-03T16:16:39.6062615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:39.6110540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-11-03T16:16:39.6130938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:39.6176025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-11-03T16:16:39.6191562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:39.6242535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-11-03T16:16:39.6255076Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:39.6300292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-11-03T16:16:39.6318173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:39.6362647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-11-03T16:16:39.6386950Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:39.6439755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-11-03T16:16:39.6459792Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:39.6508608Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-11-03T16:16:39.6530404Z Entering 'third_party/kleidiai' 2025-11-03T16:16:39.6575449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-11-03T16:16:39.6594896Z Entering 'third_party/mimalloc' 2025-11-03T16:16:39.6639614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-11-03T16:16:39.6658006Z Entering 'third_party/nlohmann' 2025-11-03T16:16:39.6702814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-11-03T16:16:39.6725741Z Entering 'third_party/onnx' 2025-11-03T16:16:39.6768257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-11-03T16:16:39.6797395Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:39.6849806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-11-03T16:16:39.6867938Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:16:39.6916928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-11-03T16:16:39.6933107Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:39.6976924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-11-03T16:16:39.6996631Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:39.7039828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-11-03T16:16:39.7060409Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:39.7103552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-11-03T16:16:39.7124577Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:39.7171009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-11-03T16:16:39.7191456Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:39.7238980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-11-03T16:16:39.7254718Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:39.7299524Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-11-03T16:16:39.7313175Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:39.7360832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-11-03T16:16:39.7376270Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:39.7424175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-11-03T16:16:39.7443701Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:39.7490712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-11-03T16:16:39.7512581Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:39.7559373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-11-03T16:16:39.7591944Z Entering 'third_party/pocketfft' 2025-11-03T16:16:39.7639167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-11-03T16:16:39.7656345Z Entering 'third_party/protobuf' 2025-11-03T16:16:39.7708595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-11-03T16:16:39.7730186Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:39.7775001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-11-03T16:16:39.7792483Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:39.7841405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-11-03T16:16:39.7863701Z Entering 'third_party/psimd' 2025-11-03T16:16:39.7909661Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-11-03T16:16:39.7928583Z Entering 'third_party/pthreadpool' 2025-11-03T16:16:39.7975593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-11-03T16:16:39.7993655Z Entering 'third_party/pybind11' 2025-11-03T16:16:39.8042402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-11-03T16:16:39.8061304Z Entering 'third_party/python-peachpy' 2025-11-03T16:16:39.8106326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-11-03T16:16:39.8126673Z Entering 'third_party/sleef' 2025-11-03T16:16:39.8170723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-11-03T16:16:39.8188629Z Entering 'third_party/tensorpipe' 2025-11-03T16:16:39.8236979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-11-03T16:16:39.8252888Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:39.8297079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-11-03T16:16:39.8314104Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:39.8359695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-11-03T16:16:39.8375741Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:39.8423195Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-11-03T16:16:39.8440382Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:39.8485758Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-11-03T16:16:39.8501996Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:39.8548654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-11-03T16:16:40.0198833Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-11-03T16:16:40.0523510Z Entering 'android/libs/fbjni' 2025-11-03T16:16:40.0562756Z Entering 'third_party/FP16' 2025-11-03T16:16:40.0601913Z Entering 'third_party/FXdiv' 2025-11-03T16:16:40.0644800Z Entering 'third_party/NNPACK' 2025-11-03T16:16:40.0686227Z Entering 'third_party/NVTX' 2025-11-03T16:16:40.0727003Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:16:40.0767777Z Entering 'third_party/XNNPACK' 2025-11-03T16:16:40.0825829Z Entering 'third_party/aiter' 2025-11-03T16:16:40.0863014Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:16:40.0909701Z Entering 'third_party/benchmark' 2025-11-03T16:16:40.0954482Z Entering 'third_party/composable_kernel' 2025-11-03T16:16:40.0996132Z Entering 'third_party/cpp-httplib' 2025-11-03T16:16:40.1039623Z Entering 'third_party/cpuinfo' 2025-11-03T16:16:40.1081192Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:16:40.1121009Z Entering 'third_party/cutlass' 2025-11-03T16:16:40.1171391Z Entering 'third_party/fbgemm' 2025-11-03T16:16:40.1211518Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:16:40.1252968Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:16:40.1298515Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:16:40.1340302Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:16:40.1380003Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:16:40.1418590Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:16:40.1460084Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:16:40.1499513Z Entering 'third_party/flash-attention' 2025-11-03T16:16:40.1537872Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:16:40.1578533Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:16:40.1628620Z Entering 'third_party/flatbuffers' 2025-11-03T16:16:40.1670636Z Entering 'third_party/fmt' 2025-11-03T16:16:40.1707754Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:16:40.1751394Z Entering 'third_party/gloo' 2025-11-03T16:16:40.1792541Z Entering 'third_party/googletest' 2025-11-03T16:16:40.1831733Z Entering 'third_party/ideep' 2025-11-03T16:16:40.1868173Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:16:40.1915625Z Entering 'third_party/ittapi' 2025-11-03T16:16:40.1959008Z Entering 'third_party/kineto' 2025-11-03T16:16:40.1995493Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:40.2032422Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:40.2074397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:40.2114388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:40.2152428Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:40.2190251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:40.2237439Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:40.2275677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:40.2313711Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:40.2357437Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:40.2395230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:40.2430553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:40.2469123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:40.2512008Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:40.2556198Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:40.2598894Z Entering 'third_party/kleidiai' 2025-11-03T16:16:40.2643933Z Entering 'third_party/mimalloc' 2025-11-03T16:16:40.2681423Z Entering 'third_party/nlohmann' 2025-11-03T16:16:40.2722328Z Entering 'third_party/onnx' 2025-11-03T16:16:40.2771389Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:40.2815887Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:16:40.2855293Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:40.2894669Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:40.2934396Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:40.2973309Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:40.3013730Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:40.3053803Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:40.3092888Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:40.3134139Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:40.3173693Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:40.3215034Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:40.3268978Z Entering 'third_party/pocketfft' 2025-11-03T16:16:40.3309816Z Entering 'third_party/protobuf' 2025-11-03T16:16:40.3353180Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:40.3391704Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:40.3432896Z Entering 'third_party/psimd' 2025-11-03T16:16:40.3471234Z Entering 'third_party/pthreadpool' 2025-11-03T16:16:40.3512074Z Entering 'third_party/pybind11' 2025-11-03T16:16:40.3552436Z Entering 'third_party/python-peachpy' 2025-11-03T16:16:40.3591833Z Entering 'third_party/sleef' 2025-11-03T16:16:40.3632581Z Entering 'third_party/tensorpipe' 2025-11-03T16:16:40.3672355Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:40.3712224Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:40.3749571Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:40.3790547Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:40.3827496Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:40.3890404Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-11-03T16:16:40.4206688Z Entering 'android/libs/fbjni' 2025-11-03T16:16:40.4248851Z Entering 'third_party/FP16' 2025-11-03T16:16:40.4287669Z Entering 'third_party/FXdiv' 2025-11-03T16:16:40.4332058Z Entering 'third_party/NNPACK' 2025-11-03T16:16:40.4373502Z Entering 'third_party/NVTX' 2025-11-03T16:16:40.4410998Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:16:40.4456426Z Entering 'third_party/XNNPACK' 2025-11-03T16:16:40.4505794Z Entering 'third_party/aiter' 2025-11-03T16:16:40.4548182Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:16:40.4593787Z Entering 'third_party/benchmark' 2025-11-03T16:16:40.4633408Z Entering 'third_party/composable_kernel' 2025-11-03T16:16:40.4683131Z Entering 'third_party/cpp-httplib' 2025-11-03T16:16:40.4725574Z Entering 'third_party/cpuinfo' 2025-11-03T16:16:40.4762527Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:16:40.4803607Z Entering 'third_party/cutlass' 2025-11-03T16:16:40.4850079Z Entering 'third_party/fbgemm' 2025-11-03T16:16:40.4890483Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:16:40.4931478Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:16:40.4976573Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:16:40.5014638Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:16:40.5059628Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:16:40.5097628Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:16:40.5139201Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:16:40.5178399Z Entering 'third_party/flash-attention' 2025-11-03T16:16:40.5223209Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:16:40.5265614Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:16:40.5313734Z Entering 'third_party/flatbuffers' 2025-11-03T16:16:40.5356232Z Entering 'third_party/fmt' 2025-11-03T16:16:40.5396213Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:16:40.5440618Z Entering 'third_party/gloo' 2025-11-03T16:16:40.5478477Z Entering 'third_party/googletest' 2025-11-03T16:16:40.5520304Z Entering 'third_party/ideep' 2025-11-03T16:16:40.5553849Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:16:40.5598489Z Entering 'third_party/ittapi' 2025-11-03T16:16:40.5642164Z Entering 'third_party/kineto' 2025-11-03T16:16:40.5680534Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:40.5716278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:40.5759096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:40.5798101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:40.5835171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:40.5870278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:40.5914773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:40.5960209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:40.5998726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:40.6039341Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:40.6080503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:40.6116385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:40.6158783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:40.6202540Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:40.6241057Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:40.6282371Z Entering 'third_party/kleidiai' 2025-11-03T16:16:40.6325742Z Entering 'third_party/mimalloc' 2025-11-03T16:16:40.6363958Z Entering 'third_party/nlohmann' 2025-11-03T16:16:40.6404346Z Entering 'third_party/onnx' 2025-11-03T16:16:40.6456764Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:40.6498574Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:16:40.6539509Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:40.6577436Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:40.6617756Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:40.6657115Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:40.6695289Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:40.6732302Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:40.6772284Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:40.6811761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:40.6849902Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:40.6892457Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:40.6947938Z Entering 'third_party/pocketfft' 2025-11-03T16:16:40.6987707Z Entering 'third_party/protobuf' 2025-11-03T16:16:40.7030121Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:40.7068259Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:40.7111356Z Entering 'third_party/psimd' 2025-11-03T16:16:40.7154254Z Entering 'third_party/pthreadpool' 2025-11-03T16:16:40.7196126Z Entering 'third_party/pybind11' 2025-11-03T16:16:40.7240153Z Entering 'third_party/python-peachpy' 2025-11-03T16:16:40.7279852Z Entering 'third_party/sleef' 2025-11-03T16:16:40.7319705Z Entering 'third_party/tensorpipe' 2025-11-03T16:16:40.7358995Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:40.7395206Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:40.7438377Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:40.7476809Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:40.7514171Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:40.7578596Z ##[endgroup] 2025-11-03T16:16:40.7608000Z [command]/usr/bin/git log -1 --format=%H 2025-11-03T16:16:40.7632455Z 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:16:40.7730090Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-11-03T16:16:40.7730340Z cd "${GITHUB_WORKSPACE}" 2025-11-03T16:16:40.7730536Z # Clean stale submodule dirs 2025-11-03T16:16:40.7730735Z if [ -z "${NO_SUDO}" ]; then 2025-11-03T16:16:40.7730959Z  sudo git submodule foreach --recursive git clean -ffdx 2025-11-03T16:16:40.7731188Z else 2025-11-03T16:16:40.7731379Z  git submodule foreach --recursive git clean -ffdx 2025-11-03T16:16:40.7731593Z fi 2025-11-03T16:16:40.7738918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:40.7739146Z env: 2025-11-03T16:16:40.7739301Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:40.7739483Z NO_SUDO: true 2025-11-03T16:16:40.7739634Z ##[endgroup] 2025-11-03T16:16:40.8070884Z Entering 'android/libs/fbjni' 2025-11-03T16:16:40.8104776Z Entering 'third_party/FP16' 2025-11-03T16:16:40.8137412Z Entering 'third_party/FXdiv' 2025-11-03T16:16:40.8168562Z Entering 'third_party/NNPACK' 2025-11-03T16:16:40.8201667Z Entering 'third_party/NVTX' 2025-11-03T16:16:40.8242148Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:16:40.8274394Z Entering 'third_party/XNNPACK' 2025-11-03T16:16:40.8375675Z Entering 'third_party/aiter' 2025-11-03T16:16:40.8410806Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:16:40.8502610Z Entering 'third_party/benchmark' 2025-11-03T16:16:40.8537912Z Entering 'third_party/composable_kernel' 2025-11-03T16:16:40.8629822Z Entering 'third_party/cpp-httplib' 2025-11-03T16:16:40.8662349Z Entering 'third_party/cpuinfo' 2025-11-03T16:16:40.8698197Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:16:40.8731482Z Entering 'third_party/cutlass' 2025-11-03T16:16:40.8809094Z Entering 'third_party/fbgemm' 2025-11-03T16:16:40.8859222Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:16:40.8887964Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:16:40.8977130Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:16:40.9009728Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:16:40.9084837Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:16:40.9115346Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:16:40.9146041Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:16:40.9187988Z Entering 'third_party/flash-attention' 2025-11-03T16:16:40.9227533Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:16:40.9304172Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:16:40.9376975Z Entering 'third_party/flatbuffers' 2025-11-03T16:16:40.9432945Z Entering 'third_party/fmt' 2025-11-03T16:16:40.9465205Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:16:40.9497895Z Entering 'third_party/gloo' 2025-11-03T16:16:40.9530775Z Entering 'third_party/googletest' 2025-11-03T16:16:40.9563337Z Entering 'third_party/ideep' 2025-11-03T16:16:40.9591110Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:16:40.9661918Z Entering 'third_party/ittapi' 2025-11-03T16:16:40.9694972Z Entering 'third_party/kineto' 2025-11-03T16:16:40.9728292Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:16:40.9761201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:16:40.9801037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:16:40.9836373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:16:40.9867240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:16:40.9896438Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:16:40.9926821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:16:40.9959065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:16:40.9988809Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:16:41.0024043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:16:41.0054192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:16:41.0085048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:41.0129845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:41.0167803Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:16:41.0204249Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:16:41.0241552Z Entering 'third_party/kleidiai' 2025-11-03T16:16:41.0278314Z Entering 'third_party/mimalloc' 2025-11-03T16:16:41.0311952Z Entering 'third_party/nlohmann' 2025-11-03T16:16:41.0353712Z Entering 'third_party/onnx' 2025-11-03T16:16:41.0571814Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:16:41.0604305Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:16:41.0650822Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:16:41.0683249Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:16:41.0711953Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:16:41.0745801Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:16:41.0783016Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:16:41.0811880Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:16:41.0844044Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:16:41.0872740Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:16:41.0913716Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:16:41.0950283Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:16:41.1134926Z Entering 'third_party/pocketfft' 2025-11-03T16:16:41.1169677Z Entering 'third_party/protobuf' 2025-11-03T16:16:41.1228980Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:16:41.1261874Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:16:41.1299499Z Entering 'third_party/psimd' 2025-11-03T16:16:41.1330142Z Entering 'third_party/pthreadpool' 2025-11-03T16:16:41.1361353Z Entering 'third_party/pybind11' 2025-11-03T16:16:41.1393474Z Entering 'third_party/python-peachpy' 2025-11-03T16:16:41.1425780Z Entering 'third_party/sleef' 2025-11-03T16:16:41.1459597Z Entering 'third_party/tensorpipe' 2025-11-03T16:16:41.1490932Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:16:41.1523587Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:16:41.1553269Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:16:41.1588302Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:16:41.1617845Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:16:41.1744412Z Prepare all required actions 2025-11-03T16:16:41.1744786Z Getting action download info 2025-11-03T16:16:41.3667890Z ##[group]Run ./.github/actions/setup-linux 2025-11-03T16:16:41.3668106Z env: 2025-11-03T16:16:41.3668259Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:41.3668428Z ##[endgroup] 2025-11-03T16:16:41.3707094Z ##[group]Run set -euo pipefail 2025-11-03T16:16:41.3707338Z set -euo pipefail 2025-11-03T16:16:41.3707529Z function get_ec2_metadata() { 2025-11-03T16:16:41.3707768Z  # Pulled from instance metadata endpoint for EC2 2025-11-03T16:16:41.3708136Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-11-03T16:16:41.3708473Z  category=$1 2025-11-03T16:16:41.3708699Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-11-03T16:16:41.3708962Z  runner_name_str=i-0d3c8af4c7ead8235 2025-11-03T16:16:41.3709205Z  if [[ -f /.inarc ]]; then 2025-11-03T16:16:41.3709414Z  echo "ARC Runner, no info on ec2 metadata" 2025-11-03T16:16:41.3709652Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-11-03T16:16:41.3709924Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-11-03T16:16:41.3710169Z  else 2025-11-03T16:16:41.3710651Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-11-03T16:16:41.3711126Z  fi 2025-11-03T16:16:41.3711271Z } 2025-11-03T16:16:41.3711447Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-11-03T16:16:41.3711712Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-11-03T16:16:41.3711987Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-11-03T16:16:41.3712234Z echo "system info $(uname -a)" 2025-11-03T16:16:41.3717318Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:41.3717546Z env: 2025-11-03T16:16:41.3717689Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:41.3717860Z ##[endgroup] 2025-11-03T16:16:41.3890970Z ami-id: ami-08982f1c5bf93d976 2025-11-03T16:16:41.3991845Z instance-id: i-0d3c8af4c7ead8235 2025-11-03T16:16:41.4083704Z instance-type: m7i-flex.8xlarge 2025-11-03T16:16:41.4094388Z system info Linux ip-10-0-34-62.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-11-03T16:16:41.4119198Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-11-03T16:16:41.4119506Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-11-03T16:16:41.4123956Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:41.4124175Z env: 2025-11-03T16:16:41.4124326Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:41.4124499Z ##[endgroup] 2025-11-03T16:16:41.4171393Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-11-03T16:16:41.4171931Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-11-03T16:16:41.4175826Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:41.4176061Z env: 2025-11-03T16:16:41.4176208Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:41.4176383Z ##[endgroup] 2025-11-03T16:16:41.4232026Z ##[group]Run if systemctl is-active --quiet docker; then 2025-11-03T16:16:41.4232309Z if systemctl is-active --quiet docker; then 2025-11-03T16:16:41.4232546Z  echo "Docker daemon is running..."; 2025-11-03T16:16:41.4232854Z else 2025-11-03T16:16:41.4233073Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-11-03T16:16:41.4233322Z fi 2025-11-03T16:16:41.4237233Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:41.4237470Z env: 2025-11-03T16:16:41.4237614Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:41.4237783Z ##[endgroup] 2025-11-03T16:16:41.4322219Z Docker daemon is running... 2025-11-03T16:16:41.4357914Z ##[group]Run nick-fields/retry@v3.0.0 2025-11-03T16:16:41.4358113Z with: 2025-11-03T16:16:41.4358250Z shell: bash 2025-11-03T16:16:41.4358407Z timeout_minutes: 5 2025-11-03T16:16:41.4358573Z max_attempts: 3 2025-11-03T16:16:41.4358737Z retry_wait_seconds: 30 2025-11-03T16:16:41.4359995Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-11-03T16:16:41.4361212Z polling_interval_seconds: 1 2025-11-03T16:16:41.4361393Z warning_on_retry: true 2025-11-03T16:16:41.4361560Z continue_on_error: false 2025-11-03T16:16:41.4361720Z env: 2025-11-03T16:16:41.4361865Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:41.4362037Z AWS_RETRY_MODE: standard 2025-11-03T16:16:41.4362203Z AWS_MAX_ATTEMPTS: 5 2025-11-03T16:16:41.4362368Z AWS_DEFAULT_REGION: us-east-1 2025-11-03T16:16:41.4362544Z ##[endgroup] 2025-11-03T16:16:42.3767435Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-11-03T16:16:42.3767875Z Configure a credential helper to remove this warning. See 2025-11-03T16:16:42.3768240Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-11-03T16:16:42.3768470Z 2025-11-03T16:16:42.3768539Z Login Succeeded 2025-11-03T16:16:42.5737423Z Command completed after 1 attempt(s). 2025-11-03T16:16:42.5802171Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-11-03T16:16:42.5802496Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-11-03T16:16:42.5802767Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-11-03T16:16:42.5808905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:42.5809129Z env: 2025-11-03T16:16:42.5809280Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:42.5809453Z ##[endgroup] 2025-11-03T16:16:42.5917744Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-11-03T16:16:42.5918096Z # ignore expansion of "docker ps -q" since it could be empty 2025-11-03T16:16:42.5918369Z # shellcheck disable=SC2046 2025-11-03T16:16:42.5918584Z docker stop $(docker ps -q) || true 2025-11-03T16:16:42.5918794Z # Prune all of the docker images 2025-11-03T16:16:42.5919008Z docker system prune -af 2025-11-03T16:16:42.5923379Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:42.5923611Z env: 2025-11-03T16:16:42.5923753Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:42.5923924Z ##[endgroup] 2025-11-03T16:16:42.6408872Z "docker stop" requires at least 1 argument. 2025-11-03T16:16:42.6409440Z See 'docker stop --help'. 2025-11-03T16:16:42.6409699Z 2025-11-03T16:16:42.6409918Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-11-03T16:16:42.6410099Z 2025-11-03T16:16:42.6410185Z Stop one or more running containers 2025-11-03T16:16:42.6575324Z Total reclaimed space: 0B 2025-11-03T16:16:42.6700934Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-11-03T16:16:42.6701309Z with: 2025-11-03T16:16:42.6701794Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6702318Z use-custom-docker-registry: true 2025-11-03T16:16:42.6702521Z docker-build-dir: .ci/docker 2025-11-03T16:16:42.6702704Z docker-build-script: ./build.sh 2025-11-03T16:16:42.6702894Z working-directory: . 2025-11-03T16:16:42.6703113Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:42.6703347Z force-push: false 2025-11-03T16:16:42.6703490Z env: 2025-11-03T16:16:42.6703631Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:42.6703796Z ##[endgroup] 2025-11-03T16:16:42.6726959Z ##[group]Run set -ex 2025-11-03T16:16:42.6727170Z set -ex 2025-11-03T16:16:42.6727325Z  2025-11-03T16:16:42.6727609Z # If the docker build directory or the build script doesn't exist, the action will 2025-11-03T16:16:42.6728000Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-11-03T16:16:42.6728322Z # job could then download the pre-built image as usual 2025-11-03T16:16:42.6728717Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-11-03T16:16:42.6729092Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6729294Z else 2025-11-03T16:16:42.6729466Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6729728Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6729969Z  2025-11-03T16:16:42.6730291Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-11-03T16:16:42.6730652Z  exit 0 2025-11-03T16:16:42.6730793Z fi 2025-11-03T16:16:42.6730937Z  2025-11-03T16:16:42.6731159Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-11-03T16:16:42.6731497Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-11-03T16:16:42.6731798Z  # use it as it is, but first let's extract the tag 2025-11-03T16:16:42.6732072Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-11-03T16:16:42.6732363Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6732639Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6732869Z else 2025-11-03T16:16:42.6733038Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-11-03T16:16:42.6733255Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-11-03T16:16:42.6733487Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-11-03T16:16:42.6733683Z  fi 2025-11-03T16:16:42.6733948Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-11-03T16:16:42.6734276Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6734622Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6735002Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6735245Z fi 2025-11-03T16:16:42.6739828Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:42.6740064Z env: 2025-11-03T16:16:42.6740215Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:42.6740391Z REPO_NAME: pytorch 2025-11-03T16:16:42.6741064Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6741591Z DOCKER_BUILD_DIR: .ci/docker 2025-11-03T16:16:42.6741862Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-11-03T16:16:42.6742119Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:42.6742364Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-11-03T16:16:42.6742543Z CUSTOM_TAG_PREFIX: 2025-11-03T16:16:42.6742705Z ##[endgroup] 2025-11-03T16:16:42.6767072Z + [[ -d .ci/docker ]] 2025-11-03T16:16:42.6768967Z + [[ -f .ci/docker/./build.sh ]] 2025-11-03T16:16:42.6769300Z + [[ true == \t\r\u\e ]] 2025-11-03T16:16:42.6771879Z + echo skip=false 2025-11-03T16:16:42.6772787Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-11-03T16:16:42.6778880Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6780095Z ++ awk -F '[:,]' '{print $2}' 2025-11-03T16:16:42.6805385Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6806148Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6807436Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6829718Z ##[group]Run set +e 2025-11-03T16:16:42.6829926Z set +e 2025-11-03T16:16:42.6830082Z set -x 2025-11-03T16:16:42.6830232Z  2025-11-03T16:16:42.6830367Z login() { 2025-11-03T16:16:42.6830664Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-11-03T16:16:42.6830982Z } 2025-11-03T16:16:42.6831122Z  2025-11-03T16:16:42.6831252Z retry () { 2025-11-03T16:16:42.6831436Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-11-03T16:16:42.6831631Z } 2025-11-03T16:16:42.6831766Z  2025-11-03T16:16:42.6831906Z retry login "${DOCKER_REGISTRY}" 2025-11-03T16:16:42.6832092Z  2025-11-03T16:16:42.6832233Z START_TIME=$(date +%s) 2025-11-03T16:16:42.6832423Z # Wait up to 120 minutes 2025-11-03T16:16:42.6832651Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-11-03T16:16:42.6832934Z  # Check if image already exists, if it does then skip building it 2025-11-03T16:16:42.6833226Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-11-03T16:16:42.6833441Z  exit 0 2025-11-03T16:16:42.6833591Z  fi 2025-11-03T16:16:42.6833723Z  2025-11-03T16:16:42.6834044Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-11-03T16:16:42.6834440Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-11-03T16:16:42.6834844Z  # latter, it will wait for the Docker images to become available before continuing 2025-11-03T16:16:42.6835156Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-11-03T16:16:42.6835400Z  # It's a Docker build job, let's build the image 2025-11-03T16:16:42.6835613Z  break 2025-11-03T16:16:42.6835769Z  else 2025-11-03T16:16:42.6835985Z  # It's a regular build job, wait for the image to become available 2025-11-03T16:16:42.6836231Z  sleep 300 2025-11-03T16:16:42.6836392Z  fi 2025-11-03T16:16:42.6836536Z done 2025-11-03T16:16:42.6836676Z  2025-11-03T16:16:42.6836883Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-11-03T16:16:42.6837300Z # be empty. The default action would be to continue rebuild the image 2025-11-03T16:16:42.6837661Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-11-03T16:16:42.6837931Z  # if we're on the base branch then use the parent commit 2025-11-03T16:16:42.6838173Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-11-03T16:16:42.6838355Z else 2025-11-03T16:16:42.6838560Z  # otherwise we're on a PR, so use the most recent base commit 2025-11-03T16:16:42.6838835Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-11-03T16:16:42.6839043Z fi 2025-11-03T16:16:42.6839172Z  2025-11-03T16:16:42.6839326Z if [[ -z "${MERGE_BASE}" ]]; then 2025-11-03T16:16:42.6839548Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6839744Z  2025-11-03T16:16:42.6840010Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-11-03T16:16:42.6840314Z  exit 0 2025-11-03T16:16:42.6840458Z fi 2025-11-03T16:16:42.6840596Z  2025-11-03T16:16:42.6840787Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-11-03T16:16:42.6841154Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-11-03T16:16:42.6841642Z  exit 1 2025-11-03T16:16:42.6841794Z fi 2025-11-03T16:16:42.6841934Z  2025-11-03T16:16:42.6842146Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-11-03T16:16:42.6842522Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-11-03T16:16:42.6842850Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-11-03T16:16:42.6843229Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-11-03T16:16:42.6843643Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-11-03T16:16:42.6843894Z fi 2025-11-03T16:16:42.6844031Z  2025-11-03T16:16:42.6844196Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-11-03T16:16:42.6848191Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:42.6848415Z env: 2025-11-03T16:16:42.6848567Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:42.6848749Z DOCKER_BUILD_DIR: .ci/docker 2025-11-03T16:16:42.6848965Z BASE_REVISION: 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:16:42.6849517Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6850198Z DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:42.6850626Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:42.6850860Z DOCKER_PUSH: 2025-11-03T16:16:42.6851007Z ##[endgroup] 2025-11-03T16:16:42.6871442Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:42.6871859Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:42.6877665Z + aws ecr get-login-password --region us-east-1 2025-11-03T16:16:42.6883080Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:43.1004877Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-11-03T16:16:43.1005225Z Login Succeeded 2025-11-03T16:16:43.1006651Z Configure a credential helper to remove this warning. See 2025-11-03T16:16:43.1007100Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-11-03T16:16:43.1011970Z 2025-11-03T16:16:43.1023415Z ++ date +%s 2025-11-03T16:16:43.1032992Z + START_TIME=1762186603 2025-11-03T16:16:43.1055108Z ++ date +%s 2025-11-03T16:16:43.1059809Z + [[ 1762179403 -lt 1762186603 ]] 2025-11-03T16:16:43.1062615Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:43.3247520Z { 2025-11-03T16:16:43.3249190Z "schemaVersion": 2, 2025-11-03T16:16:43.3249620Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-11-03T16:16:43.3252521Z "config": { 2025-11-03T16:16:43.3252910Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-11-03T16:16:43.3257453Z "size": 31771, 2025-11-03T16:16:43.3260505Z "digest": "sha256:a0defb4d3830e67aa1d3adb1251bb453436eb10d09880acf2cba4d790dae265b" 2025-11-03T16:16:43.3262610Z }, 2025-11-03T16:16:43.3262902Z "layers": [ 2025-11-03T16:16:43.3265699Z + exit 0 2025-11-03T16:16:43.3265881Z { 2025-11-03T16:16:43.3266130Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3266399Z "size": 30447990, 2025-11-03T16:16:43.3266710Z "digest": "sha256:828c1365039a657352c737a62d13e1932951b5658eb6bd9b9096ea9b73562453" 2025-11-03T16:16:43.3267003Z }, 2025-11-03T16:16:43.3267141Z { 2025-11-03T16:16:43.3267447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3267719Z "size": 1552, 2025-11-03T16:16:43.3268003Z "digest": "sha256:1c4ee0d43392f96acf52d2e8e2f6257611a1334051823ff62d69c216dbc7daca" 2025-11-03T16:16:43.3268349Z }, 2025-11-03T16:16:43.3268482Z { 2025-11-03T16:16:43.3268689Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3268973Z "size": 313625489, 2025-11-03T16:16:43.3269250Z "digest": "sha256:903f2614f01d7b5d60c33432bb448f286bd7baf86973c272f37f5caf3a83407b" 2025-11-03T16:16:43.3269532Z }, 2025-11-03T16:16:43.3269658Z { 2025-11-03T16:16:43.3269872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3270127Z "size": 791, 2025-11-03T16:16:43.3270398Z "digest": "sha256:9a925ed696bfcc8ebda91a3c220252d69635e24adf4fc2835afd1a4aa9e601f6" 2025-11-03T16:16:43.3270679Z }, 2025-11-03T16:16:43.3270811Z { 2025-11-03T16:16:43.3271023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3271278Z "size": 106, 2025-11-03T16:16:43.3271537Z "digest": "sha256:d91c445dab95bd9421e1967175f55dfeca3f30166b80aa76eb6bae7b72718d65" 2025-11-03T16:16:43.3271826Z }, 2025-11-03T16:16:43.3272061Z { 2025-11-03T16:16:43.3272272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3272527Z "size": 703, 2025-11-03T16:16:43.3272781Z "digest": "sha256:33a0c5e93563ea73282e573325b1527d301f618dbec76bb05c07228cf1c69230" 2025-11-03T16:16:43.3273053Z }, 2025-11-03T16:16:43.3273185Z { 2025-11-03T16:16:43.3273396Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3273650Z "size": 1217, 2025-11-03T16:16:43.3273905Z "digest": "sha256:760a865b8cf6a0302880ce187235b551859954571432c6f70cfde5d89ed44572" 2025-11-03T16:16:43.3274340Z }, 2025-11-03T16:16:43.3274486Z { 2025-11-03T16:16:43.3274720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3274980Z "size": 484, 2025-11-03T16:16:43.3275263Z "digest": "sha256:9d9c7be466f87846f48df125fcf7b25bc5d633cefbf0584ed645d3ab90c043bd" 2025-11-03T16:16:43.3275570Z }, 2025-11-03T16:16:43.3275704Z { 2025-11-03T16:16:43.3275906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3276169Z "size": 110362290, 2025-11-03T16:16:43.3276445Z "digest": "sha256:8b2beb2cf1cd0a9540475610f709b97b857e883c66c0f7278d8aa2d729c0a307" 2025-11-03T16:16:43.3276732Z }, 2025-11-03T16:16:43.3276855Z { 2025-11-03T16:16:43.3277064Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3277325Z "size": 5132, 2025-11-03T16:16:43.3277588Z "digest": "sha256:6e5bdc9e922cccd285cb3e3e8710db5d3119b375d23f7f6b154436adf6f92133" 2025-11-03T16:16:43.3278109Z }, 2025-11-03T16:16:43.3278248Z { 2025-11-03T16:16:43.3278459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3278802Z "size": 1755, 2025-11-03T16:16:43.3279057Z "digest": "sha256:95bb0abcaee9af0c3aafb8021c45ce9381c0a1e2c8b4c6a394a49b98107f8102" 2025-11-03T16:16:43.3279342Z }, 2025-11-03T16:16:43.3279472Z { 2025-11-03T16:16:43.3279671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3279909Z "size": 724, 2025-11-03T16:16:43.3280164Z "digest": "sha256:b3a429f4faee83e3d7e616cbcca1b458319d213977b4807ea863fc416abb2229" 2025-11-03T16:16:43.3280442Z }, 2025-11-03T16:16:43.3280570Z { 2025-11-03T16:16:43.3280764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3281012Z "size": 544, 2025-11-03T16:16:43.3281265Z "digest": "sha256:f4cb10b91a16ac44a669c09f1037854ceb9884ac4c6a118da74679c3f2629f24" 2025-11-03T16:16:43.3281537Z }, 2025-11-03T16:16:43.3281661Z { 2025-11-03T16:16:43.3281863Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3282114Z "size": 3376456481, 2025-11-03T16:16:43.3282385Z "digest": "sha256:e3f3290c6b5078b355af4289c6d4bbd488e0e0f271e6ed456c923ecc2ae3f847" 2025-11-03T16:16:43.3282654Z }, 2025-11-03T16:16:43.3282786Z { 2025-11-03T16:16:43.3282990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3283236Z "size": 32, 2025-11-03T16:16:43.3283486Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3283768Z }, 2025-11-03T16:16:43.3283900Z { 2025-11-03T16:16:43.3284115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3284355Z "size": 380, 2025-11-03T16:16:43.3284611Z "digest": "sha256:2ec7ba999b860bc831c8e5b4e42a1087c8cfde1d02d8f2b3b0a1a297def702b1" 2025-11-03T16:16:43.3284895Z }, 2025-11-03T16:16:43.3285023Z { 2025-11-03T16:16:43.3285222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3285475Z "size": 237238, 2025-11-03T16:16:43.3285728Z "digest": "sha256:44205544068086409131b0bb441b6bae62601e637f5e046f07b3db50f22bca6b" 2025-11-03T16:16:43.3286001Z }, 2025-11-03T16:16:43.3286123Z { 2025-11-03T16:16:43.3286325Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3286575Z "size": 230, 2025-11-03T16:16:43.3286829Z "digest": "sha256:df8f8108c158cd5519b0ea50bd22522f354b19bf2388184e23faa3733b3fd6e3" 2025-11-03T16:16:43.3287098Z }, 2025-11-03T16:16:43.3287229Z { 2025-11-03T16:16:43.3287438Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3287677Z "size": 3561745, 2025-11-03T16:16:43.3287918Z "digest": "sha256:611e20e3c292b27c7b577e741c97d251ccffe119e097ef0fe40cda74c29c9898" 2025-11-03T16:16:43.3288182Z }, 2025-11-03T16:16:43.3288305Z { 2025-11-03T16:16:43.3288500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3288730Z "size": 1478, 2025-11-03T16:16:43.3288974Z "digest": "sha256:397c5718353c9270969b965bfddacd659394a4189be78c230558a674cbde1a2c" 2025-11-03T16:16:43.3289231Z }, 2025-11-03T16:16:43.3289353Z { 2025-11-03T16:16:43.3289538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3289770Z "size": 481, 2025-11-03T16:16:43.3290005Z "digest": "sha256:099139c9d8b455920b3e71991a0af82a00315f2708e1b8398089dad4889d0c82" 2025-11-03T16:16:43.3290263Z }, 2025-11-03T16:16:43.3290377Z { 2025-11-03T16:16:43.3290571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3290804Z "size": 198, 2025-11-03T16:16:43.3291050Z "digest": "sha256:68bdc7025dfdee58119a7d1fe7cfcca7e26e099fe86b1d71eb8e7d255bdf9a02" 2025-11-03T16:16:43.3291319Z }, 2025-11-03T16:16:43.3291438Z { 2025-11-03T16:16:43.3291631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3291915Z "size": 607, 2025-11-03T16:16:43.3292160Z "digest": "sha256:a99b38a3dcdd8fe005f4f28ca62e73a4c465ac67ea320f9cfee2fcae35797245" 2025-11-03T16:16:43.3292476Z }, 2025-11-03T16:16:43.3292601Z { 2025-11-03T16:16:43.3292798Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3293028Z "size": 226, 2025-11-03T16:16:43.3293270Z "digest": "sha256:dde94b02539f4c5c0fa5f0fe7656c218e33469860c5cf5fa8bca26b165f6c9df" 2025-11-03T16:16:43.3293543Z }, 2025-11-03T16:16:43.3293672Z { 2025-11-03T16:16:43.3293865Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3294102Z "size": 830, 2025-11-03T16:16:43.3294341Z "digest": "sha256:1a6a81cb899b0f9a55045ca3f89823d22816308be9440327b64990ae8108de42" 2025-11-03T16:16:43.3294604Z }, 2025-11-03T16:16:43.3294729Z { 2025-11-03T16:16:43.3294918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3295155Z "size": 32, 2025-11-03T16:16:43.3295403Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3295675Z }, 2025-11-03T16:16:43.3295789Z { 2025-11-03T16:16:43.3295985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3296227Z "size": 104, 2025-11-03T16:16:43.3296470Z "digest": "sha256:87f538f29435cfd8b415415ce225b9c93299fe3ca99eeeea4d3bee114dde63f2" 2025-11-03T16:16:43.3296731Z }, 2025-11-03T16:16:43.3296855Z { 2025-11-03T16:16:43.3297052Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3297289Z "size": 1496, 2025-11-03T16:16:43.3297527Z "digest": "sha256:5f740b394ea310b0c2f28c055d3ba824427d5b2b55a9226ac4039be54b839698" 2025-11-03T16:16:43.3297792Z }, 2025-11-03T16:16:43.3297917Z { 2025-11-03T16:16:43.3298110Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3298342Z "size": 458156451, 2025-11-03T16:16:43.3298601Z "digest": "sha256:d8a01999be1b874eca3a842515bc821323412c2dbdbaf3a29d1fe694e2524fd6" 2025-11-03T16:16:43.3298868Z }, 2025-11-03T16:16:43.3298992Z { 2025-11-03T16:16:43.3299180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3299415Z "size": 163, 2025-11-03T16:16:43.3299658Z "digest": "sha256:5b782dd120ccdf33aaf9e37cc92912234658ad8e357ad324250a3f63044177d6" 2025-11-03T16:16:43.3299924Z }, 2025-11-03T16:16:43.3300039Z { 2025-11-03T16:16:43.3300235Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3300473Z "size": 346, 2025-11-03T16:16:43.3300716Z "digest": "sha256:c084e8e1076b1f971aad4348e38c6204c20c0d3b768959f05a5f7e298c600d46" 2025-11-03T16:16:43.3300983Z }, 2025-11-03T16:16:43.3301105Z { 2025-11-03T16:16:43.3301301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3301537Z "size": 32, 2025-11-03T16:16:43.3301775Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3302046Z }, 2025-11-03T16:16:43.3302168Z { 2025-11-03T16:16:43.3302368Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3302595Z "size": 106, 2025-11-03T16:16:43.3302837Z "digest": "sha256:69ac68a6fed1f9351916b290d069db2d34f9729590b892a7878c4124ad171daa" 2025-11-03T16:16:43.3303100Z }, 2025-11-03T16:16:43.3303222Z { 2025-11-03T16:16:43.3303412Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3303648Z "size": 425, 2025-11-03T16:16:43.3303889Z "digest": "sha256:6345aec6eb298d06b276130fd8c432205e01f3090bf30bce281256fb484b6f4c" 2025-11-03T16:16:43.3304155Z }, 2025-11-03T16:16:43.3304268Z { 2025-11-03T16:16:43.3304464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3304698Z "size": 19309387, 2025-11-03T16:16:43.3304946Z "digest": "sha256:f70b7643328788a61482695ed5366eea113121b02a10e13b9fdfb849a3917460" 2025-11-03T16:16:43.3305241Z }, 2025-11-03T16:16:43.3305366Z { 2025-11-03T16:16:43.3305562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3305834Z "size": 108, 2025-11-03T16:16:43.3306071Z "digest": "sha256:c489de58735c98d8fdb49756ef3652f545687e849db5581d5eeb55d9b6ce2309" 2025-11-03T16:16:43.3306337Z }, 2025-11-03T16:16:43.3306466Z { 2025-11-03T16:16:43.3306666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3306899Z "size": 639, 2025-11-03T16:16:43.3307142Z "digest": "sha256:14f0993d5468420d7d0c2454a7b63ebd7a0a1c8a27674982e3a4427f49ee1b4c" 2025-11-03T16:16:43.3307409Z }, 2025-11-03T16:16:43.3307541Z { 2025-11-03T16:16:43.3307738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3307983Z "size": 724, 2025-11-03T16:16:43.3308240Z "digest": "sha256:b3a429f4faee83e3d7e616cbcca1b458319d213977b4807ea863fc416abb2229" 2025-11-03T16:16:43.3308521Z }, 2025-11-03T16:16:43.3308646Z { 2025-11-03T16:16:43.3308860Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3309114Z "size": 149, 2025-11-03T16:16:43.3309364Z "digest": "sha256:d6a558a7de5e9dd5633745ba04365d261f0c5ef255d68335a1e34d2393acabd6" 2025-11-03T16:16:43.3309633Z }, 2025-11-03T16:16:43.3309765Z { 2025-11-03T16:16:43.3309971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3310215Z "size": 135, 2025-11-03T16:16:43.3310461Z "digest": "sha256:05ee86267141af52f7acaa389053961b9b3af580b4cfa52635fb11c3cc9ac77a" 2025-11-03T16:16:43.3310741Z }, 2025-11-03T16:16:43.3310874Z { 2025-11-03T16:16:43.3311078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3311315Z "size": 140, 2025-11-03T16:16:43.3311574Z "digest": "sha256:f0ccf17afa340e506e78617ef552a5e1bcde2d3a8f329d0741997cabdd68b3df" 2025-11-03T16:16:43.3311858Z }, 2025-11-03T16:16:43.3311988Z { 2025-11-03T16:16:43.3312186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3312437Z "size": 18917582702, 2025-11-03T16:16:43.3312700Z "digest": "sha256:8e344790570220fe72790c8888398552e561ea57f254ec6e09b58c05918ea948" 2025-11-03T16:16:43.3312967Z }, 2025-11-03T16:16:43.3313090Z { 2025-11-03T16:16:43.3313570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3313837Z "size": 223, 2025-11-03T16:16:43.3314207Z "digest": "sha256:68d52eeaeab5519dece6ba7184103c08523328e46c08acd06ccde7574f53c99f" 2025-11-03T16:16:43.3314521Z }, 2025-11-03T16:16:43.3314662Z { 2025-11-03T16:16:43.3314888Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3315150Z "size": 255, 2025-11-03T16:16:43.3315422Z "digest": "sha256:1ef3a6930902936a8a4d2f964a465c0b4ceecec2fd03a78c0e5b4a4174a6dcc7" 2025-11-03T16:16:43.3315697Z }, 2025-11-03T16:16:43.3315826Z { 2025-11-03T16:16:43.3316030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3316268Z "size": 32, 2025-11-03T16:16:43.3316518Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3316794Z }, 2025-11-03T16:16:43.3316919Z { 2025-11-03T16:16:43.3317110Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3317351Z "size": 106, 2025-11-03T16:16:43.3317594Z "digest": "sha256:e3d341ef61f7422b033c83df566da087b0227d5451f1cc476ea23e7b2b7740c8" 2025-11-03T16:16:43.3317865Z }, 2025-11-03T16:16:43.3317984Z { 2025-11-03T16:16:43.3318180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3318422Z "size": 312300627, 2025-11-03T16:16:43.3318676Z "digest": "sha256:f02010d8bae596b2708ac7564b74946b89c210df33b7002237adf157e8c8155e" 2025-11-03T16:16:43.3318937Z }, 2025-11-03T16:16:43.3319064Z { 2025-11-03T16:16:43.3319263Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3319592Z "size": 6655935991, 2025-11-03T16:16:43.3319857Z "digest": "sha256:e1b8e0ed540895fded01aeeed90e057b04e8578a0fe7ab03a2eae459d3b4f3d2" 2025-11-03T16:16:43.3320215Z }, 2025-11-03T16:16:43.3320350Z { 2025-11-03T16:16:43.3320556Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3320790Z "size": 129, 2025-11-03T16:16:43.3321041Z "digest": "sha256:d570696e5d553ae86e0adb52c5936a58c19ba372cfe6beb38f38b77f5528813e" 2025-11-03T16:16:43.3321314Z }, 2025-11-03T16:16:43.3321438Z { 2025-11-03T16:16:43.3321630Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3321871Z "size": 880, 2025-11-03T16:16:43.3322111Z "digest": "sha256:2a86c078c73d1d0224e1046a3e2ca94663ca04652ec9ac44700e7fd6a4d9fc9c" 2025-11-03T16:16:43.3322378Z }, 2025-11-03T16:16:43.3322496Z { 2025-11-03T16:16:43.3322692Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3322932Z "size": 724, 2025-11-03T16:16:43.3323179Z "digest": "sha256:b3a429f4faee83e3d7e616cbcca1b458319d213977b4807ea863fc416abb2229" 2025-11-03T16:16:43.3323445Z }, 2025-11-03T16:16:43.3323570Z { 2025-11-03T16:16:43.3323769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3324004Z "size": 139, 2025-11-03T16:16:43.3324241Z "digest": "sha256:44e5c47c9ce4024f40e82cfb276df42771be0a6991ba260f37a9789f3caee04a" 2025-11-03T16:16:43.3324508Z }, 2025-11-03T16:16:43.3324632Z { 2025-11-03T16:16:43.3324829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3325059Z "size": 32, 2025-11-03T16:16:43.3325311Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3325574Z }, 2025-11-03T16:16:43.3325694Z { 2025-11-03T16:16:43.3325881Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3326113Z "size": 158, 2025-11-03T16:16:43.3326345Z "digest": "sha256:64251afee5607817a662f32274265243a89150f093f443332e98aea6edf4b8c5" 2025-11-03T16:16:43.3326599Z }, 2025-11-03T16:16:43.3326717Z { 2025-11-03T16:16:43.3326907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3327139Z "size": 1011, 2025-11-03T16:16:43.3327375Z "digest": "sha256:07c8d78a73e65a5453a1045bb24c0080b003ffc3168f56f16fea4c30f9581f5d" 2025-11-03T16:16:43.3327624Z }, 2025-11-03T16:16:43.3327743Z { 2025-11-03T16:16:43.3327937Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3328167Z "size": 724, 2025-11-03T16:16:43.3328398Z "digest": "sha256:b3a429f4faee83e3d7e616cbcca1b458319d213977b4807ea863fc416abb2229" 2025-11-03T16:16:43.3328663Z }, 2025-11-03T16:16:43.3328785Z { 2025-11-03T16:16:43.3328978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3329203Z "size": 134, 2025-11-03T16:16:43.3329437Z "digest": "sha256:66372f032f924ed974c345878c60ae95400e3826d05cadb854b5b378c86fef10" 2025-11-03T16:16:43.3329698Z }, 2025-11-03T16:16:43.3329822Z { 2025-11-03T16:16:43.3330012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3330253Z "size": 32, 2025-11-03T16:16:43.3330495Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3330756Z }, 2025-11-03T16:16:43.3330870Z { 2025-11-03T16:16:43.3331061Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3331290Z "size": 158, 2025-11-03T16:16:43.3331540Z "digest": "sha256:6ccf69de9ffaa1f7e6cd2ddebd9f98af1d94abfd70314be5eecc82dc4891666b" 2025-11-03T16:16:43.3331812Z }, 2025-11-03T16:16:43.3331928Z { 2025-11-03T16:16:43.3332120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3332355Z "size": 1371, 2025-11-03T16:16:43.3332595Z "digest": "sha256:b41ffa5a6a4588148bf64c1e6d509af9d49b9ff7f27f4f7be606e4c0a48f3a12" 2025-11-03T16:16:43.3332861Z }, 2025-11-03T16:16:43.3333031Z { 2025-11-03T16:16:43.3333225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3333499Z "size": 32, 2025-11-03T16:16:43.3333739Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3334006Z }, 2025-11-03T16:16:43.3334130Z { 2025-11-03T16:16:43.3334319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3334558Z "size": 136, 2025-11-03T16:16:43.3334797Z "digest": "sha256:9387991f0cf43904780f0c45bca4aae99172554d189ae6f23c98f53e81a04426" 2025-11-03T16:16:43.3335058Z }, 2025-11-03T16:16:43.3335180Z { 2025-11-03T16:16:43.3335365Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3335598Z "size": 528, 2025-11-03T16:16:43.3335841Z "digest": "sha256:16ec736149b0011270995bb0bab5befaafbc716cfdf9f280ff25f75a2072c90d" 2025-11-03T16:16:43.3336106Z }, 2025-11-03T16:16:43.3336218Z { 2025-11-03T16:16:43.3336415Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3336651Z "size": 32, 2025-11-03T16:16:43.3336891Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3337150Z }, 2025-11-03T16:16:43.3337271Z { 2025-11-03T16:16:43.3337465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3337699Z "size": 104, 2025-11-03T16:16:43.3337933Z "digest": "sha256:c747e0bc263fe0d17a44b7d4c74d6425b6f2217bd55902965a6cc04bdeda73ad" 2025-11-03T16:16:43.3338200Z }, 2025-11-03T16:16:43.3338324Z { 2025-11-03T16:16:43.3338517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3338748Z "size": 428, 2025-11-03T16:16:43.3338986Z "digest": "sha256:b294669af2b5786b9bc5601bffdc874e4673b0fcb879e1d77c17e674e7954b10" 2025-11-03T16:16:43.3339255Z }, 2025-11-03T16:16:43.3339377Z { 2025-11-03T16:16:43.3339567Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3339805Z "size": 32, 2025-11-03T16:16:43.3340052Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3340323Z }, 2025-11-03T16:16:43.3340437Z { 2025-11-03T16:16:43.3340632Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3340865Z "size": 109, 2025-11-03T16:16:43.3341113Z "digest": "sha256:1374eac977ffde8aecf0963fc3b932faabb747b62e49bd3fbbc18a70d542ac07" 2025-11-03T16:16:43.3341387Z }, 2025-11-03T16:16:43.3341512Z { 2025-11-03T16:16:43.3341710Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3341947Z "size": 1896, 2025-11-03T16:16:43.3342187Z "digest": "sha256:375f6cf74921eec948c69077e8280fd9d9c2a6ccf83f8a33cccc413fc36edca8" 2025-11-03T16:16:43.3342458Z }, 2025-11-03T16:16:43.3342579Z { 2025-11-03T16:16:43.3342775Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3343010Z "size": 245087916, 2025-11-03T16:16:43.3343269Z "digest": "sha256:858a1f759af33b3ca72bdb0ffba6ab64f9cd7369c2ac929156aa92ee260cde57" 2025-11-03T16:16:43.3343545Z }, 2025-11-03T16:16:43.3343688Z { 2025-11-03T16:16:43.3343875Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3344111Z "size": 106, 2025-11-03T16:16:43.3344350Z "digest": "sha256:46276d66d3e318a1a2c1f38b45b5060a30068f3b486d0d71e58fd249b7e10687" 2025-11-03T16:16:43.3344611Z }, 2025-11-03T16:16:43.3344725Z { 2025-11-03T16:16:43.3344921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3345154Z "size": 165, 2025-11-03T16:16:43.3345390Z "digest": "sha256:83ecc1143684a6535b14ce79395dec432eca66768055598312e14a7d4522ad8b" 2025-11-03T16:16:43.3345645Z }, 2025-11-03T16:16:43.3345770Z { 2025-11-03T16:16:43.3345966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3346203Z "size": 7944, 2025-11-03T16:16:43.3346481Z "digest": "sha256:84d7e8a5d95e595a910304a466fdf855f4ffc34878881b1e1a3c9124207fdb5d" 2025-11-03T16:16:43.3346786Z }, 2025-11-03T16:16:43.3346910Z { 2025-11-03T16:16:43.3347101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3347329Z "size": 8076, 2025-11-03T16:16:43.3347575Z "digest": "sha256:ee91858ea3c4ed12a1a712424c09d95f407c78ceeb31ec21d1874220fa773ce4" 2025-11-03T16:16:43.3347845Z }, 2025-11-03T16:16:43.3347969Z { 2025-11-03T16:16:43.3348156Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3348390Z "size": 303, 2025-11-03T16:16:43.3348624Z "digest": "sha256:a5d33870f153d309e9c3150f196979fdaee17378d91b1ed8257bad66194c99c6" 2025-11-03T16:16:43.3348883Z }, 2025-11-03T16:16:43.3349000Z { 2025-11-03T16:16:43.3349194Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3349427Z "size": 32, 2025-11-03T16:16:43.3349671Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3349933Z }, 2025-11-03T16:16:43.3350060Z { 2025-11-03T16:16:43.3350258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3350490Z "size": 108, 2025-11-03T16:16:43.3350724Z "digest": "sha256:5af10c4c6ef231eeebee585067812841135c727daab1183fd6a6b244a186cd34" 2025-11-03T16:16:43.3350990Z }, 2025-11-03T16:16:43.3351115Z { 2025-11-03T16:16:43.3351309Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3351538Z "size": 54145660, 2025-11-03T16:16:43.3351791Z "digest": "sha256:6823ad83408ec2d1d628bf5e7cd49d5f6a990ea8001fed41ae77c958e655c8ae" 2025-11-03T16:16:43.3352058Z }, 2025-11-03T16:16:43.3352184Z { 2025-11-03T16:16:43.3352372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-11-03T16:16:43.3352607Z "size": 32, 2025-11-03T16:16:43.3352850Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-11-03T16:16:43.3353117Z } 2025-11-03T16:16:43.3353238Z ] 2025-11-03T16:16:43.3353371Z } 2025-11-03T16:16:43.3381897Z ##[group]Run set -eux 2025-11-03T16:16:43.3382092Z set -eux 2025-11-03T16:16:43.3382351Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-11-03T16:16:43.3382998Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-11-03T16:16:43.3387909Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:43.3388129Z env: 2025-11-03T16:16:43.3388274Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:43.3388445Z ##[endgroup] 2025-11-03T16:16:43.3413743Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-11-03T16:16:43.3414066Z + jq --raw-output .SecretString 2025-11-03T16:16:43.3414327Z + docker login --username pytorchbot --password-stdin 2025-11-03T16:16:43.3420151Z + jq -r .docker_hub_readonly_token 2025-11-03T16:16:43.7876177Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-11-03T16:16:43.7876533Z Login Succeeded 2025-11-03T16:16:43.7877038Z Configure a credential helper to remove this warning. See 2025-11-03T16:16:43.7877408Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-11-03T16:16:43.7877641Z 2025-11-03T16:16:43.7959639Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-11-03T16:16:43.7959893Z tag=${ECR_DOCKER_IMAGE##*:} 2025-11-03T16:16:43.7960151Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-11-03T16:16:43.7964867Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:43.7965092Z env: 2025-11-03T16:16:43.7965239Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:43.7965761Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:43.7966371Z ##[endgroup] 2025-11-03T16:16:43.7991206Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:43.8033216Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-11-03T16:16:43.8033474Z with: 2025-11-03T16:16:43.8033943Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:43.8034659Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:43.8034924Z env: 2025-11-03T16:16:43.8035087Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:43.8035277Z ##[endgroup] 2025-11-03T16:16:43.8052212Z ##[group]Run set -x 2025-11-03T16:16:43.8052409Z set -x 2025-11-03T16:16:43.8052555Z set +e 2025-11-03T16:16:43.8052699Z  2025-11-03T16:16:43.8052841Z login() { 2025-11-03T16:16:43.8053132Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-11-03T16:16:43.8053443Z } 2025-11-03T16:16:43.8053575Z  2025-11-03T16:16:43.8053744Z retry () { 2025-11-03T16:16:43.8053919Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-11-03T16:16:43.8054102Z } 2025-11-03T16:16:43.8054236Z  2025-11-03T16:16:43.8054388Z retry login "${DOCKER_REGISTRY}" 2025-11-03T16:16:43.8054577Z  2025-11-03T16:16:43.8054851Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-11-03T16:16:43.8055221Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-11-03T16:16:43.8055439Z  2025-11-03T16:16:43.8055570Z set -e 2025-11-03T16:16:43.8055772Z # ignore output since only exit code is used for conditional 2025-11-03T16:16:43.8056052Z # only pull docker image if it's not available locally 2025-11-03T16:16:43.8056365Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-11-03T16:16:43.8056652Z  retry docker pull "${DOCKER_IMAGE}" 2025-11-03T16:16:43.8056843Z fi 2025-11-03T16:16:43.8060586Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:16:43.8060808Z env: 2025-11-03T16:16:43.8060955Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:16:43.8061440Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:43.8061979Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:43.8062208Z ##[endgroup] 2025-11-03T16:16:43.8083065Z + set +e 2025-11-03T16:16:43.8088547Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:43.8090499Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:43.8090952Z + aws ecr get-login-password --region us-east-1 2025-11-03T16:16:43.8091425Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-11-03T16:16:44.2226447Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-11-03T16:16:44.2226784Z Login Succeeded 2025-11-03T16:16:44.2228687Z Configure a credential helper to remove this warning. See 2025-11-03T16:16:44.2229265Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-11-03T16:16:44.2229603Z 2025-11-03T16:16:44.2251607Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:44.2253809Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-11-03T16:16:44.4648222Z + IMAGE_SIZE=29084.48217201233 2025-11-03T16:16:44.4648559Z Compressed size of image in MB: 29084.48217201233 2025-11-03T16:16:44.4652782Z + echo 'Compressed size of image in MB: 29084.48217201233' 2025-11-03T16:16:44.4657958Z + set -e 2025-11-03T16:16:44.4663213Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:44.4793429Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:44.4794577Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:16:44.7089775Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c: Pulling from pytorch/ci-image 2025-11-03T16:16:44.7094185Z 828c1365039a: Pulling fs layer 2025-11-03T16:16:44.7096018Z 1c4ee0d43392: Pulling fs layer 2025-11-03T16:16:44.7096477Z 903f2614f01d: Pulling fs layer 2025-11-03T16:16:44.7096750Z 9a925ed696bf: Pulling fs layer 2025-11-03T16:16:44.7097012Z d91c445dab95: Pulling fs layer 2025-11-03T16:16:44.7097235Z 33a0c5e93563: Pulling fs layer 2025-11-03T16:16:44.7097467Z 760a865b8cf6: Pulling fs layer 2025-11-03T16:16:44.7097701Z 9d9c7be466f8: Pulling fs layer 2025-11-03T16:16:44.7097908Z 8b2beb2cf1cd: Pulling fs layer 2025-11-03T16:16:44.7098113Z 6e5bdc9e922c: Pulling fs layer 2025-11-03T16:16:44.7098325Z 95bb0abcaee9: Pulling fs layer 2025-11-03T16:16:44.7098539Z b3a429f4faee: Pulling fs layer 2025-11-03T16:16:44.7098747Z f4cb10b91a16: Pulling fs layer 2025-11-03T16:16:44.7098942Z e3f3290c6b50: Pulling fs layer 2025-11-03T16:16:44.7099110Z 4f4fb700ef54: Pulling fs layer 2025-11-03T16:16:44.7099288Z 2ec7ba999b86: Pulling fs layer 2025-11-03T16:16:44.7099468Z 442055440680: Pulling fs layer 2025-11-03T16:16:44.7099663Z df8f8108c158: Pulling fs layer 2025-11-03T16:16:44.7099849Z 611e20e3c292: Pulling fs layer 2025-11-03T16:16:44.7100077Z 397c5718353c: Pulling fs layer 2025-11-03T16:16:44.7100271Z 099139c9d8b4: Pulling fs layer 2025-11-03T16:16:44.7100499Z 68bdc7025dfd: Pulling fs layer 2025-11-03T16:16:44.7100693Z a99b38a3dcdd: Pulling fs layer 2025-11-03T16:16:44.7100992Z dde94b02539f: Pulling fs layer 2025-11-03T16:16:44.7101747Z 1a6a81cb899b: Pulling fs layer 2025-11-03T16:16:44.7102019Z 87f538f29435: Pulling fs layer 2025-11-03T16:16:44.7102312Z 5f740b394ea3: Pulling fs layer 2025-11-03T16:16:44.7102520Z d8a01999be1b: Pulling fs layer 2025-11-03T16:16:44.7102703Z 5b782dd120cc: Pulling fs layer 2025-11-03T16:16:44.7102881Z c084e8e1076b: Pulling fs layer 2025-11-03T16:16:44.7103130Z 69ac68a6fed1: Pulling fs layer 2025-11-03T16:16:44.7109282Z 6345aec6eb29: Pulling fs layer 2025-11-03T16:16:44.7111487Z f70b76433287: Pulling fs layer 2025-11-03T16:16:44.7111846Z c489de58735c: Pulling fs layer 2025-11-03T16:16:44.7118008Z 14f0993d5468: Pulling fs layer 2025-11-03T16:16:44.7122102Z d6a558a7de5e: Pulling fs layer 2025-11-03T16:16:44.7126295Z 05ee86267141: Pulling fs layer 2025-11-03T16:16:44.7130898Z f0ccf17afa34: Pulling fs layer 2025-11-03T16:16:44.7135089Z 8e3447905702: Pulling fs layer 2025-11-03T16:16:44.7137136Z 68d52eeaeab5: Pulling fs layer 2025-11-03T16:16:44.7137357Z 1ef3a6930902: Pulling fs layer 2025-11-03T16:16:44.7137552Z e3d341ef61f7: Pulling fs layer 2025-11-03T16:16:44.7137732Z f02010d8bae5: Pulling fs layer 2025-11-03T16:16:44.7137915Z e1b8e0ed5408: Pulling fs layer 2025-11-03T16:16:44.7138092Z d570696e5d55: Pulling fs layer 2025-11-03T16:16:44.7138265Z 2a86c078c73d: Pulling fs layer 2025-11-03T16:16:44.7138437Z 44e5c47c9ce4: Pulling fs layer 2025-11-03T16:16:44.7138610Z 64251afee560: Pulling fs layer 2025-11-03T16:16:44.7138788Z 07c8d78a73e6: Pulling fs layer 2025-11-03T16:16:44.7138962Z 66372f032f92: Pulling fs layer 2025-11-03T16:16:44.7139190Z 6ccf69de9ffa: Pulling fs layer 2025-11-03T16:16:44.7139360Z b41ffa5a6a45: Pulling fs layer 2025-11-03T16:16:44.7139532Z 9387991f0cf4: Pulling fs layer 2025-11-03T16:16:44.7139703Z 16ec736149b0: Pulling fs layer 2025-11-03T16:16:44.7139875Z c747e0bc263f: Pulling fs layer 2025-11-03T16:16:44.7140293Z b294669af2b5: Pulling fs layer 2025-11-03T16:16:44.7140471Z 1374eac977ff: Pulling fs layer 2025-11-03T16:16:44.7140745Z 375f6cf74921: Pulling fs layer 2025-11-03T16:16:44.7140923Z 858a1f759af3: Pulling fs layer 2025-11-03T16:16:44.7141090Z 46276d66d3e3: Pulling fs layer 2025-11-03T16:16:44.7141261Z 83ecc1143684: Pulling fs layer 2025-11-03T16:16:44.7141431Z 84d7e8a5d95e: Pulling fs layer 2025-11-03T16:16:44.7141606Z ee91858ea3c4: Pulling fs layer 2025-11-03T16:16:44.7141773Z a5d33870f153: Pulling fs layer 2025-11-03T16:16:44.7141947Z 5af10c4c6ef2: Pulling fs layer 2025-11-03T16:16:44.7142121Z 6823ad83408e: Pulling fs layer 2025-11-03T16:16:44.7142292Z 9d9c7be466f8: Waiting 2025-11-03T16:16:44.7142449Z 8b2beb2cf1cd: Waiting 2025-11-03T16:16:44.7142603Z 6e5bdc9e922c: Waiting 2025-11-03T16:16:44.7142755Z 95bb0abcaee9: Waiting 2025-11-03T16:16:44.7142902Z b3a429f4faee: Waiting 2025-11-03T16:16:44.7143054Z f4cb10b91a16: Waiting 2025-11-03T16:16:44.7143205Z e3f3290c6b50: Waiting 2025-11-03T16:16:44.7143356Z 4f4fb700ef54: Waiting 2025-11-03T16:16:44.7143497Z 2ec7ba999b86: Waiting 2025-11-03T16:16:44.7143643Z 442055440680: Waiting 2025-11-03T16:16:44.7143792Z df8f8108c158: Waiting 2025-11-03T16:16:44.7143939Z 611e20e3c292: Waiting 2025-11-03T16:16:44.7144110Z 099139c9d8b4: Waiting 2025-11-03T16:16:44.7144253Z 68bdc7025dfd: Waiting 2025-11-03T16:16:44.7144404Z a99b38a3dcdd: Waiting 2025-11-03T16:16:44.7144551Z 397c5718353c: Waiting 2025-11-03T16:16:44.7144697Z dde94b02539f: Waiting 2025-11-03T16:16:44.7144840Z 1a6a81cb899b: Waiting 2025-11-03T16:16:44.7144990Z 69ac68a6fed1: Waiting 2025-11-03T16:16:44.7145138Z 87f538f29435: Waiting 2025-11-03T16:16:44.7145286Z 6345aec6eb29: Waiting 2025-11-03T16:16:44.7145427Z d8a01999be1b: Waiting 2025-11-03T16:16:44.7145576Z 5b782dd120cc: Waiting 2025-11-03T16:16:44.7145725Z f70b76433287: Waiting 2025-11-03T16:16:44.7145875Z c489de58735c: Waiting 2025-11-03T16:16:44.7146016Z 14f0993d5468: Waiting 2025-11-03T16:16:44.7146165Z c084e8e1076b: Waiting 2025-11-03T16:16:44.7146320Z d6a558a7de5e: Waiting 2025-11-03T16:16:44.7146468Z 05ee86267141: Waiting 2025-11-03T16:16:44.7146609Z f0ccf17afa34: Waiting 2025-11-03T16:16:44.7146758Z 8e3447905702: Waiting 2025-11-03T16:16:44.7146907Z 68d52eeaeab5: Waiting 2025-11-03T16:16:44.7147057Z 1ef3a6930902: Waiting 2025-11-03T16:16:44.7147199Z 16ec736149b0: Waiting 2025-11-03T16:16:44.7147347Z 64251afee560: Waiting 2025-11-03T16:16:44.7147496Z 07c8d78a73e6: Waiting 2025-11-03T16:16:44.7147644Z c747e0bc263f: Waiting 2025-11-03T16:16:44.7147784Z 66372f032f92: Waiting 2025-11-03T16:16:44.7147931Z b294669af2b5: Waiting 2025-11-03T16:16:44.7148077Z 1374eac977ff: Waiting 2025-11-03T16:16:44.7148218Z e3d341ef61f7: Waiting 2025-11-03T16:16:44.7148368Z 375f6cf74921: Waiting 2025-11-03T16:16:44.7148517Z f02010d8bae5: Waiting 2025-11-03T16:16:44.7148664Z 858a1f759af3: Waiting 2025-11-03T16:16:44.7148806Z e1b8e0ed5408: Waiting 2025-11-03T16:16:44.7148954Z 2a86c078c73d: Waiting 2025-11-03T16:16:44.7149101Z 46276d66d3e3: Waiting 2025-11-03T16:16:44.7149252Z 83ecc1143684: Waiting 2025-11-03T16:16:44.7149394Z 9a925ed696bf: Waiting 2025-11-03T16:16:44.7149542Z d570696e5d55: Waiting 2025-11-03T16:16:44.7149691Z 44e5c47c9ce4: Waiting 2025-11-03T16:16:44.7149839Z 6ccf69de9ffa: Waiting 2025-11-03T16:16:44.7149982Z ee91858ea3c4: Waiting 2025-11-03T16:16:44.7150132Z d91c445dab95: Waiting 2025-11-03T16:16:44.7150282Z a5d33870f153: Waiting 2025-11-03T16:16:44.7150428Z 6823ad83408e: Waiting 2025-11-03T16:16:44.7150571Z 5af10c4c6ef2: Waiting 2025-11-03T16:16:44.7150721Z 9387991f0cf4: Waiting 2025-11-03T16:16:44.7150873Z b41ffa5a6a45: Waiting 2025-11-03T16:16:44.7151024Z 33a0c5e93563: Waiting 2025-11-03T16:16:44.7151166Z 760a865b8cf6: Waiting 2025-11-03T16:16:44.7151315Z 84d7e8a5d95e: Waiting 2025-11-03T16:16:44.7775682Z 1c4ee0d43392: Verifying Checksum 2025-11-03T16:16:44.7778673Z 1c4ee0d43392: Download complete 2025-11-03T16:16:44.8669748Z 9a925ed696bf: Verifying Checksum 2025-11-03T16:16:44.8670230Z 9a925ed696bf: Download complete 2025-11-03T16:16:44.9486700Z d91c445dab95: Download complete 2025-11-03T16:16:45.0147503Z 33a0c5e93563: Verifying Checksum 2025-11-03T16:16:45.0151891Z 33a0c5e93563: Download complete 2025-11-03T16:16:45.0709073Z 828c1365039a: Verifying Checksum 2025-11-03T16:16:45.0714398Z 828c1365039a: Download complete 2025-11-03T16:16:45.0947717Z 760a865b8cf6: Verifying Checksum 2025-11-03T16:16:45.0948247Z 760a865b8cf6: Download complete 2025-11-03T16:16:45.1488423Z 9d9c7be466f8: Verifying Checksum 2025-11-03T16:16:45.1488923Z 9d9c7be466f8: Download complete 2025-11-03T16:16:45.2209008Z 6e5bdc9e922c: Verifying Checksum 2025-11-03T16:16:45.2209518Z 6e5bdc9e922c: Download complete 2025-11-03T16:16:45.2942999Z 95bb0abcaee9: Verifying Checksum 2025-11-03T16:16:45.2943387Z 95bb0abcaee9: Download complete 2025-11-03T16:16:45.3845389Z b3a429f4faee: Verifying Checksum 2025-11-03T16:16:45.3845734Z b3a429f4faee: Download complete 2025-11-03T16:16:45.4955563Z f4cb10b91a16: Verifying Checksum 2025-11-03T16:16:45.4956130Z f4cb10b91a16: Download complete 2025-11-03T16:16:46.1445284Z 828c1365039a: Pull complete 2025-11-03T16:16:46.1599840Z 1c4ee0d43392: Pull complete 2025-11-03T16:16:46.2525848Z 8b2beb2cf1cd: Verifying Checksum 2025-11-03T16:16:46.2529117Z 8b2beb2cf1cd: Download complete 2025-11-03T16:16:46.2604238Z 4f4fb700ef54: Verifying Checksum 2025-11-03T16:16:46.2607242Z 4f4fb700ef54: Download complete 2025-11-03T16:16:46.3629928Z 2ec7ba999b86: Verifying Checksum 2025-11-03T16:16:46.3632071Z 2ec7ba999b86: Download complete 2025-11-03T16:16:46.4618874Z 442055440680: Verifying Checksum 2025-11-03T16:16:46.4623684Z 442055440680: Download complete 2025-11-03T16:16:46.5367980Z df8f8108c158: Verifying Checksum 2025-11-03T16:16:46.5370048Z df8f8108c158: Download complete 2025-11-03T16:16:46.6465717Z 611e20e3c292: Verifying Checksum 2025-11-03T16:16:46.6465999Z 611e20e3c292: Download complete 2025-11-03T16:16:46.7040930Z 397c5718353c: Download complete 2025-11-03T16:16:46.7784347Z 099139c9d8b4: Verifying Checksum 2025-11-03T16:16:46.7784799Z 099139c9d8b4: Download complete 2025-11-03T16:16:46.8714872Z 68bdc7025dfd: Download complete 2025-11-03T16:16:46.9734083Z a99b38a3dcdd: Verifying Checksum 2025-11-03T16:16:46.9735815Z a99b38a3dcdd: Download complete 2025-11-03T16:16:47.0413603Z dde94b02539f: Verifying Checksum 2025-11-03T16:16:47.0413881Z dde94b02539f: Download complete 2025-11-03T16:16:47.1215453Z 1a6a81cb899b: Verifying Checksum 2025-11-03T16:16:47.1219417Z 1a6a81cb899b: Download complete 2025-11-03T16:16:47.2014924Z 87f538f29435: Verifying Checksum 2025-11-03T16:16:47.2015408Z 87f538f29435: Download complete 2025-11-03T16:16:47.2837564Z 5f740b394ea3: Verifying Checksum 2025-11-03T16:16:47.2840167Z 5f740b394ea3: Download complete 2025-11-03T16:16:47.9051117Z 903f2614f01d: Verifying Checksum 2025-11-03T16:16:47.9055154Z 903f2614f01d: Download complete 2025-11-03T16:16:47.9766232Z 5b782dd120cc: Verifying Checksum 2025-11-03T16:16:47.9771275Z 5b782dd120cc: Download complete 2025-11-03T16:16:48.0541910Z c084e8e1076b: Verifying Checksum 2025-11-03T16:16:48.0548153Z c084e8e1076b: Download complete 2025-11-03T16:16:48.1147159Z 69ac68a6fed1: Verifying Checksum 2025-11-03T16:16:48.1153012Z 69ac68a6fed1: Download complete 2025-11-03T16:16:48.4612183Z f70b76433287: Verifying Checksum 2025-11-03T16:16:48.4613090Z f70b76433287: Download complete 2025-11-03T16:16:48.5359422Z c489de58735c: Verifying Checksum 2025-11-03T16:16:48.5361247Z c489de58735c: Download complete 2025-11-03T16:16:48.6427576Z 14f0993d5468: Download complete 2025-11-03T16:16:48.7109603Z d6a558a7de5e: Verifying Checksum 2025-11-03T16:16:48.7109914Z d6a558a7de5e: Download complete 2025-11-03T16:16:48.8065746Z 05ee86267141: Verifying Checksum 2025-11-03T16:16:48.8066255Z 05ee86267141: Download complete 2025-11-03T16:16:48.8661228Z f0ccf17afa34: Verifying Checksum 2025-11-03T16:16:48.8663141Z f0ccf17afa34: Download complete 2025-11-03T16:16:51.9380363Z d8a01999be1b: Verifying Checksum 2025-11-03T16:16:51.9380684Z d8a01999be1b: Download complete 2025-11-03T16:16:52.0185637Z 68d52eeaeab5: Verifying Checksum 2025-11-03T16:16:52.0191206Z 68d52eeaeab5: Download complete 2025-11-03T16:16:52.1008212Z 1ef3a6930902: Verifying Checksum 2025-11-03T16:16:52.1012943Z 1ef3a6930902: Download complete 2025-11-03T16:16:52.1798745Z e3d341ef61f7: Verifying Checksum 2025-11-03T16:16:52.1800394Z e3d341ef61f7: Download complete 2025-11-03T16:16:55.3650694Z f02010d8bae5: Verifying Checksum 2025-11-03T16:16:55.3651005Z f02010d8bae5: Download complete 2025-11-03T16:16:59.2709205Z 903f2614f01d: Pull complete 2025-11-03T16:16:59.6301019Z 9a925ed696bf: Pull complete 2025-11-03T16:16:59.9273940Z d91c445dab95: Pull complete 2025-11-03T16:17:00.2292133Z 33a0c5e93563: Pull complete 2025-11-03T16:17:00.5991503Z 760a865b8cf6: Pull complete 2025-11-03T16:17:00.8900129Z 9d9c7be466f8: Pull complete 2025-11-03T16:17:04.4145396Z 8b2beb2cf1cd: Pull complete 2025-11-03T16:17:04.6747245Z 6e5bdc9e922c: Pull complete 2025-11-03T16:17:04.9805424Z 95bb0abcaee9: Pull complete 2025-11-03T16:17:05.1909227Z b3a429f4faee: Pull complete 2025-11-03T16:17:05.3865927Z f4cb10b91a16: Pull complete 2025-11-03T16:17:19.3569472Z e3f3290c6b50: Verifying Checksum 2025-11-03T16:17:19.3570967Z e3f3290c6b50: Download complete 2025-11-03T16:17:19.4550975Z d570696e5d55: Verifying Checksum 2025-11-03T16:17:19.4551552Z d570696e5d55: Download complete 2025-11-03T16:17:19.5238901Z 2a86c078c73d: Verifying Checksum 2025-11-03T16:17:19.5244078Z 2a86c078c73d: Download complete 2025-11-03T16:17:19.6161168Z 44e5c47c9ce4: Verifying Checksum 2025-11-03T16:17:19.6162892Z 44e5c47c9ce4: Download complete 2025-11-03T16:17:19.6873258Z 64251afee560: Verifying Checksum 2025-11-03T16:17:19.6875282Z 64251afee560: Download complete 2025-11-03T16:17:19.7856347Z 07c8d78a73e6: Download complete 2025-11-03T16:17:19.8767471Z 66372f032f92: Verifying Checksum 2025-11-03T16:17:19.8771946Z 66372f032f92: Download complete 2025-11-03T16:17:19.9620293Z 6ccf69de9ffa: Verifying Checksum 2025-11-03T16:17:19.9624956Z 6ccf69de9ffa: Download complete 2025-11-03T16:17:20.0331087Z b41ffa5a6a45: Verifying Checksum 2025-11-03T16:17:20.0336308Z b41ffa5a6a45: Download complete 2025-11-03T16:17:20.1145611Z 9387991f0cf4: Download complete 2025-11-03T16:17:20.2037847Z 16ec736149b0: Download complete 2025-11-03T16:17:20.3115553Z c747e0bc263f: Download complete 2025-11-03T16:17:20.3845717Z b294669af2b5: Verifying Checksum 2025-11-03T16:17:20.4725177Z 1374eac977ff: Verifying Checksum 2025-11-03T16:17:20.4728917Z 1374eac977ff: Download complete 2025-11-03T16:17:20.5594617Z 375f6cf74921: Verifying Checksum 2025-11-03T16:17:20.5594933Z 375f6cf74921: Download complete 2025-11-03T16:17:23.0611783Z 858a1f759af3: Verifying Checksum 2025-11-03T16:17:23.0617015Z 858a1f759af3: Download complete 2025-11-03T16:17:23.1366060Z 46276d66d3e3: Verifying Checksum 2025-11-03T16:17:23.1369881Z 46276d66d3e3: Download complete 2025-11-03T16:17:23.1968323Z 83ecc1143684: Verifying Checksum 2025-11-03T16:17:23.1971218Z 83ecc1143684: Download complete 2025-11-03T16:17:23.2694029Z 84d7e8a5d95e: Download complete 2025-11-03T16:17:23.3497642Z ee91858ea3c4: Verifying Checksum 2025-11-03T16:17:23.3497962Z ee91858ea3c4: Download complete 2025-11-03T16:17:23.4206279Z a5d33870f153: Verifying Checksum 2025-11-03T16:17:23.4210577Z a5d33870f153: Download complete 2025-11-03T16:17:23.4987946Z 5af10c4c6ef2: Verifying Checksum 2025-11-03T16:17:23.4989821Z 5af10c4c6ef2: Download complete 2025-11-03T16:17:24.0905039Z 6823ad83408e: Verifying Checksum 2025-11-03T16:17:24.0908303Z 6823ad83408e: Download complete 2025-11-03T16:18:01.9961828Z e1b8e0ed5408: Verifying Checksum 2025-11-03T16:18:01.9968321Z e1b8e0ed5408: Download complete 2025-11-03T16:18:36.3971162Z e3f3290c6b50: Pull complete 2025-11-03T16:18:36.6327252Z 4f4fb700ef54: Pull complete 2025-11-03T16:18:36.8605473Z 2ec7ba999b86: Pull complete 2025-11-03T16:18:37.1408359Z 442055440680: Pull complete 2025-11-03T16:18:37.4525398Z df8f8108c158: Pull complete 2025-11-03T16:18:37.8920284Z 611e20e3c292: Pull complete 2025-11-03T16:18:38.3269004Z 397c5718353c: Pull complete 2025-11-03T16:18:38.8609433Z 099139c9d8b4: Pull complete 2025-11-03T16:18:39.1759207Z 68bdc7025dfd: Pull complete 2025-11-03T16:18:39.4452350Z a99b38a3dcdd: Pull complete 2025-11-03T16:18:39.7194497Z dde94b02539f: Pull complete 2025-11-03T16:18:40.0133508Z 1a6a81cb899b: Pull complete 2025-11-03T16:18:40.4067608Z 87f538f29435: Pull complete 2025-11-03T16:18:40.6863248Z 5f740b394ea3: Pull complete 2025-11-03T16:18:51.8267645Z d8a01999be1b: Pull complete 2025-11-03T16:18:52.1791439Z 5b782dd120cc: Pull complete 2025-11-03T16:18:52.6907949Z c084e8e1076b: Pull complete 2025-11-03T16:18:53.5914992Z 69ac68a6fed1: Pull complete 2025-11-03T16:18:53.9261560Z 6345aec6eb29: Pull complete 2025-11-03T16:18:54.5474949Z f70b76433287: Pull complete 2025-11-03T16:18:54.9802425Z c489de58735c: Pull complete 2025-11-03T16:18:55.4610173Z 14f0993d5468: Pull complete 2025-11-03T16:18:56.3054173Z d6a558a7de5e: Pull complete 2025-11-03T16:18:56.8352784Z 05ee86267141: Pull complete 2025-11-03T16:18:57.3612475Z f0ccf17afa34: Pull complete 2025-11-03T16:19:58.1187026Z 8e3447905702: Download complete 2025-11-03T16:24:16.1698821Z 8e3447905702: Pull complete 2025-11-03T16:24:16.6466305Z 68d52eeaeab5: Pull complete 2025-11-03T16:24:17.0961526Z 1ef3a6930902: Pull complete 2025-11-03T16:24:18.0730785Z e3d341ef61f7: Pull complete 2025-11-03T16:24:21.2775967Z f02010d8bae5: Pull complete 2025-11-03T16:26:52.0901870Z e1b8e0ed5408: Pull complete 2025-11-03T16:26:52.1187292Z d570696e5d55: Pull complete 2025-11-03T16:26:52.1478971Z 2a86c078c73d: Pull complete 2025-11-03T16:26:52.2089142Z 44e5c47c9ce4: Pull complete 2025-11-03T16:26:52.2682597Z 64251afee560: Pull complete 2025-11-03T16:26:52.2963106Z 07c8d78a73e6: Pull complete 2025-11-03T16:26:52.3545687Z 66372f032f92: Pull complete 2025-11-03T16:26:52.4100576Z 6ccf69de9ffa: Pull complete 2025-11-03T16:26:52.4392464Z b41ffa5a6a45: Pull complete 2025-11-03T16:26:52.4945032Z 9387991f0cf4: Pull complete 2025-11-03T16:26:52.5208317Z 16ec736149b0: Pull complete 2025-11-03T16:26:52.5736675Z c747e0bc263f: Pull complete 2025-11-03T16:26:52.6012181Z b294669af2b5: Pull complete 2025-11-03T16:26:52.6548366Z 1374eac977ff: Pull complete 2025-11-03T16:26:52.6826446Z 375f6cf74921: Pull complete 2025-11-03T16:27:01.6917304Z 858a1f759af3: Pull complete 2025-11-03T16:27:02.0164707Z 46276d66d3e3: Pull complete 2025-11-03T16:27:02.3892697Z 83ecc1143684: Pull complete 2025-11-03T16:27:02.9141332Z 84d7e8a5d95e: Pull complete 2025-11-03T16:27:03.4503277Z ee91858ea3c4: Pull complete 2025-11-03T16:27:03.9763422Z a5d33870f153: Pull complete 2025-11-03T16:27:04.9104252Z 5af10c4c6ef2: Pull complete 2025-11-03T16:27:07.4933402Z 6823ad83408e: Pull complete 2025-11-03T16:27:08.1173786Z Digest: sha256:882a19f24dba11c5445e0a3a1cb4ef199b1740a1ba310ace5991cb025e30be3b 2025-11-03T16:27:08.2102257Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:27:08.2564031Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:27:08.2618686Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-11-03T16:27:08.2619253Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-11-03T16:27:08.2626612Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:08.2626841Z env: 2025-11-03T16:27:08.2626987Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:08.2627157Z ##[endgroup] 2025-11-03T16:27:08.2703085Z Prepare all required actions 2025-11-03T16:27:08.2919705Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-11-03T16:27:08.2919980Z with: 2025-11-03T16:27:08.2920581Z github-token: *** 2025-11-03T16:27:08.2920735Z env: 2025-11-03T16:27:08.2920886Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:08.2921058Z ##[endgroup] 2025-11-03T16:27:08.3040411Z ##[group]Run set -eux 2025-11-03T16:27:08.3040803Z set -eux 2025-11-03T16:27:08.3041100Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-11-03T16:27:08.3046262Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:08.3046486Z env: 2025-11-03T16:27:08.3046631Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:08.3046981Z GITHUB_TOKEN: *** 2025-11-03T16:27:08.3047135Z ##[endgroup] 2025-11-03T16:27:08.3072887Z + python3 .github/scripts/get_workflow_job_id.py 19040285537 i-0d3c8af4c7ead8235 2025-11-03T16:27:09.3239858Z Setting output job-id=54378387707 2025-11-03T16:27:09.3244339Z Setting output job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:09.3426659Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-11-03T16:27:09.3427093Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-11-03T16:27:09.3427642Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-11-03T16:27:09.3428116Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:27:09.3433493Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:09.3433725Z env: 2025-11-03T16:27:09.3433882Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:09.3434187Z JOB_ID: 54378387707 2025-11-03T16:27:09.3434471Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:09.3434775Z WORKFLOW_NAME: inductor 2025-11-03T16:27:09.3434951Z WORKFLOW_RUN_ID: 19040285537 2025-11-03T16:27:09.3435152Z MONITOR_LOG_INTERVAL: 5 2025-11-03T16:27:09.3435340Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-11-03T16:27:09.3435530Z ##[endgroup] 2025-11-03T16:27:09.8914478Z Defaulting to user installation because normal site-packages is not writeable 2025-11-03T16:27:10.1478667Z Collecting psutil==5.9.8 2025-11-03T16:27:10.1609672Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-11-03T16:27:10.2440506Z Collecting dataclasses_json==0.6.7 2025-11-03T16:27:10.2481187Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-11-03T16:27:10.3258965Z Collecting nvidia-ml-py==11.525.84 2025-11-03T16:27:10.3293955Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-11-03T16:27:10.4133584Z Collecting typing-inspect<1,>=0.4.0 2025-11-03T16:27:10.4167519Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-11-03T16:27:10.5812934Z Collecting marshmallow<4.0.0,>=3.18.0 2025-11-03T16:27:10.5844503Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-11-03T16:27:10.7122035Z Collecting packaging>=17.0 2025-11-03T16:27:10.7158476Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-11-03T16:27:10.8031506Z Collecting mypy-extensions>=0.3.0 2025-11-03T16:27:10.8069326Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-11-03T16:27:10.9202452Z Collecting typing-extensions>=3.7.4 2025-11-03T16:27:10.9237363Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-11-03T16:27:11.1584085Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-11-03T16:27:11.7130268Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-11-03T16:27:11.9600545Z Prepare all required actions 2025-11-03T16:27:11.9600837Z Getting action download info 2025-11-03T16:27:12.1475054Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-11-03T16:27:12.5655041Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-11-03T16:27:15.4232287Z ##[group]Run ./.github/actions/download-build-artifacts 2025-11-03T16:27:15.4232522Z with: 2025-11-03T16:27:15.4232694Z name: linux-jammy-py3.10-gcc11-build 2025-11-03T16:27:15.4232893Z s3-bucket: gha-artifacts 2025-11-03T16:27:15.4233065Z env: 2025-11-03T16:27:15.4233208Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:15.4233372Z ##[endgroup] 2025-11-03T16:27:15.4306888Z ##[group]Run seemethere/download-artifact-s3@v4 2025-11-03T16:27:15.4307105Z with: 2025-11-03T16:27:15.4307274Z name: linux-jammy-py3.10-gcc11-build 2025-11-03T16:27:15.4307477Z s3-bucket: gha-artifacts 2025-11-03T16:27:15.4307677Z region: us-east-1 2025-11-03T16:27:15.4307819Z env: 2025-11-03T16:27:15.4307963Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:15.4308127Z ##[endgroup] 2025-11-03T16:27:16.1303463Z (node:48467) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-11-03T16:27:16.1304760Z 2025-11-03T16:27:16.1305053Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-11-03T16:27:16.1305453Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-11-03T16:27:16.1305785Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-11-03T16:27:17.6733168Z Found 1 objects with prefix pytorch/pytorch/19040285537/linux-jammy-py3.10-gcc11-build/ 2025-11-03T16:27:17.6737301Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-11-03T16:27:25.4059167Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-11-03T16:27:25.4063061Z Artifact download has finished successfully 2025-11-03T16:27:25.4285860Z ##[group]Run unzip -o artifacts.zip 2025-11-03T16:27:25.4286099Z unzip -o artifacts.zip 2025-11-03T16:27:25.4291261Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:25.4291489Z env: 2025-11-03T16:27:25.4291640Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:25.4291812Z ##[endgroup] 2025-11-03T16:27:25.4363879Z Archive: artifacts.zip 2025-11-03T16:27:25.4368512Z creating: dist/ 2025-11-03T16:27:26.4823193Z inflating: dist/torch-2.10.0a0+git3f6538f-cp310-cp310-linux_x86_64.whl 2025-11-03T16:27:26.4824566Z creating: dist/vision/ 2025-11-03T16:27:26.4898396Z inflating: dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl 2025-11-03T16:27:26.4902616Z creating: dist/audio/ 2025-11-03T16:27:26.4924601Z inflating: dist/audio/torchaudio-2.10.0a0+3b0e7a6-cp310-cp310-linux_x86_64.whl 2025-11-03T16:27:26.4929042Z creating: dist/ao/ 2025-11-03T16:27:26.4960235Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-11-03T16:27:26.5067118Z inflating: dist/.ninja_log 2025-11-03T16:27:26.5069132Z creating: build/custom_test_artifacts/ 2025-11-03T16:27:26.5069525Z creating: build/custom_test_artifacts/custom-op-build/ 2025-11-03T16:27:26.5074142Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-11-03T16:27:26.5076110Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-11-03T16:27:26.5076700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-11-03T16:27:26.5081734Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-11-03T16:27:26.5083668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-11-03T16:27:26.5084247Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-11-03T16:27:26.5085024Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-11-03T16:27:26.5088618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-11-03T16:27:26.5089153Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-11-03T16:27:26.5089603Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-11-03T16:27:26.5090272Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-11-03T16:27:26.5090691Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-11-03T16:27:26.5091167Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-11-03T16:27:26.5091663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-11-03T16:27:26.5092119Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-11-03T16:27:26.5092600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-11-03T16:27:26.5093117Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-11-03T16:27:26.5093564Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-11-03T16:27:26.5093956Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-11-03T16:27:26.5094350Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-11-03T16:27:26.5094748Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-11-03T16:27:26.5095205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-11-03T16:27:26.5095641Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-11-03T16:27:26.5096049Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-11-03T16:27:26.5096466Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-11-03T16:27:26.5096884Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-11-03T16:27:26.5097313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-11-03T16:27:26.5097732Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-11-03T16:27:26.5098152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-11-03T16:27:26.5103101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-11-03T16:27:26.5269173Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-11-03T16:27:26.5273775Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-11-03T16:27:26.5275697Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-11-03T16:27:26.5276359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-11-03T16:27:26.5280898Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-11-03T16:27:26.5282724Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-11-03T16:27:26.5283380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-11-03T16:27:26.5286437Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-11-03T16:27:26.5287376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-11-03T16:27:26.5287877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-11-03T16:27:26.5293205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-11-03T16:27:26.5298081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-11-03T16:27:26.5357651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-11-03T16:27:26.5361980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-11-03T16:27:26.5364543Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-11-03T16:27:26.5365103Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-11-03T16:27:26.5370224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-11-03T16:27:26.5374637Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-11-03T16:27:26.5376358Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-11-03T16:27:26.5376888Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-11-03T16:27:26.5377327Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-11-03T16:27:26.5507763Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-11-03T16:27:26.5555047Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-11-03T16:27:26.5557714Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-11-03T16:27:26.5558154Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-11-03T16:27:26.5563244Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-11-03T16:27:26.5567355Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-11-03T16:27:26.5572663Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-11-03T16:27:26.5577500Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-11-03T16:27:26.5582699Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-11-03T16:27:26.5583287Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-11-03T16:27:26.5588578Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-11-03T16:27:26.5589229Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-11-03T16:27:26.5589796Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-11-03T16:27:26.5590238Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-11-03T16:27:26.5590641Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-11-03T16:27:26.5591121Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-11-03T16:27:26.5591595Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-11-03T16:27:26.5592033Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-11-03T16:27:26.5592506Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-11-03T16:27:26.5593008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-11-03T16:27:26.5593448Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-11-03T16:27:26.5594193Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-11-03T16:27:26.5594622Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-11-03T16:27:26.5595083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-11-03T16:27:26.5595598Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-11-03T16:27:26.5596153Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-11-03T16:27:26.5596586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-11-03T16:27:26.5597027Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-11-03T16:27:26.5597480Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-11-03T16:27:26.5597934Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-11-03T16:27:26.5598382Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-11-03T16:27:26.5598824Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-11-03T16:27:26.5599295Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-11-03T16:27:26.5640799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-11-03T16:27:26.5641326Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-11-03T16:27:26.5646855Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-11-03T16:27:26.5651436Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-11-03T16:27:26.5656023Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-11-03T16:27:26.5658327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-11-03T16:27:26.5664308Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-11-03T16:27:26.5669119Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-11-03T16:27:26.5669671Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-11-03T16:27:26.5679165Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-11-03T16:27:26.5681428Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-11-03T16:27:26.5686416Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-11-03T16:27:26.5688508Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-11-03T16:27:26.5689084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-11-03T16:27:26.5689597Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-11-03T16:27:26.5693413Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-11-03T16:27:26.5693925Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-11-03T16:27:26.5694354Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-11-03T16:27:26.5694849Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-11-03T16:27:26.5695337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-11-03T16:27:26.5695788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-11-03T16:27:26.5696229Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-11-03T16:27:26.5697078Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-11-03T16:27:26.5697568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-11-03T16:27:26.5698068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-11-03T16:27:26.5698527Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-11-03T16:27:26.5699086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-11-03T16:27:26.5699621Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-11-03T16:27:26.5700085Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-11-03T16:27:26.5700488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-11-03T16:27:26.5700905Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-11-03T16:27:26.5701377Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-11-03T16:27:26.5701881Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-11-03T16:27:26.5702362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-11-03T16:27:26.5702815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-11-03T16:27:26.5703287Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-11-03T16:27:26.5703758Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-11-03T16:27:26.5704231Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-11-03T16:27:26.5704695Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-11-03T16:27:26.5705157Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-11-03T16:27:26.5705657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-11-03T16:27:26.5799739Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-11-03T16:27:26.5804192Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-11-03T16:27:26.5806754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-11-03T16:27:26.5807450Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-11-03T16:27:26.5811125Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-11-03T16:27:26.5815175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-11-03T16:27:26.5817084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-11-03T16:27:26.5817756Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-11-03T16:27:26.5822771Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-11-03T16:27:26.5827198Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-11-03T16:27:26.5831315Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-11-03T16:27:26.5832118Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-11-03T16:27:26.5865207Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-11-03T16:27:26.5867444Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-11-03T16:27:26.5868371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-11-03T16:27:26.5869144Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-11-03T16:27:26.5871910Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-11-03T16:27:26.5872395Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-11-03T16:27:26.5877283Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-11-03T16:27:26.5882044Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-11-03T16:27:26.5888006Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-11-03T16:27:26.5954651Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-11-03T16:27:26.5988720Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-11-03T16:27:26.5991082Z creating: build/lib/ 2025-11-03T16:27:26.6059464Z inflating: build/lib/libprotobuf-lite.a 2025-11-03T16:27:26.6439612Z inflating: build/lib/libprotobuf.a 2025-11-03T16:27:26.6864460Z inflating: build/lib/libprotoc.a 2025-11-03T16:27:26.6871597Z inflating: build/lib/libpthreadpool.a 2025-11-03T16:27:26.6879147Z inflating: build/lib/libcpuinfo.a 2025-11-03T16:27:26.6885949Z inflating: build/lib/libcpuinfo_internals.a 2025-11-03T16:27:26.6890402Z inflating: build/lib/libclog.a 2025-11-03T16:27:26.6902134Z inflating: build/lib/libpytorch_qnnpack.a 2025-11-03T16:27:26.6906726Z inflating: build/lib/libnnpack_reference_layers.a 2025-11-03T16:27:26.7065327Z inflating: build/lib/libmicrokernels-prod.a 2025-11-03T16:27:26.7080473Z inflating: build/lib/libnnpack.a 2025-11-03T16:27:26.7840144Z inflating: build/lib/libmicrokernels-all.a 2025-11-03T16:27:26.7899483Z inflating: build/lib/libgtest.a 2025-11-03T16:27:26.7912983Z inflating: build/lib/libgmock.a 2025-11-03T16:27:26.7917578Z inflating: build/lib/libgmock_main.a 2025-11-03T16:27:26.7919692Z inflating: build/lib/libgtest_main.a 2025-11-03T16:27:26.7993200Z inflating: build/lib/libXNNPACK.a 2025-11-03T16:27:26.8058740Z inflating: build/lib/libbenchmark.a 2025-11-03T16:27:26.8063926Z inflating: build/lib/libbenchmark_main.a 2025-11-03T16:27:26.8066578Z inflating: build/lib/libjitprofiling.a 2025-11-03T16:27:26.8066946Z inflating: build/lib/libittnotify.a 2025-11-03T16:27:26.8125081Z inflating: build/lib/libasmjit.a 2025-11-03T16:27:26.9121033Z inflating: build/lib/libfbgemm.a 2025-11-03T16:27:26.9146806Z inflating: build/lib/libtensorpipe_uv.a 2025-11-03T16:27:26.9617914Z inflating: build/lib/libtensorpipe.a 2025-11-03T16:27:26.9722735Z inflating: build/lib/libgloo.a 2025-11-03T16:27:26.9760249Z inflating: build/lib/libonnx_proto.a 2025-11-03T16:27:27.0373291Z inflating: build/lib/libonnx.a 2025-11-03T16:27:27.9004744Z inflating: build/lib/libdnnl.a 2025-11-03T16:27:27.9021372Z inflating: build/lib/libfmt.a 2025-11-03T16:27:27.9260779Z inflating: build/lib/libkineto.a 2025-11-03T16:27:27.9364780Z inflating: build/lib/libc10.so 2025-11-03T16:27:27.9365190Z inflating: build/lib/libtorch_global_deps.so 2025-11-03T16:27:30.6010523Z inflating: build/lib/libtorch_cpu.so 2025-11-03T16:27:30.6011020Z inflating: build/lib/libtorch.so 2025-11-03T16:27:30.6070269Z inflating: build/lib/libtorchbind_test.so 2025-11-03T16:27:30.6088677Z inflating: build/lib/libjitbackend_test.so 2025-11-03T16:27:30.6109131Z inflating: build/lib/libbackend_with_compiler.so 2025-11-03T16:27:30.6131997Z inflating: build/lib/libaoti_custom_ops.so 2025-11-03T16:27:30.6135704Z inflating: build/lib/libshm.so 2025-11-03T16:27:30.8106779Z inflating: build/lib/libtorch_python.so 2025-11-03T16:27:30.8137882Z inflating: build/lib/libnnapi_backend.so 2025-11-03T16:27:30.8138575Z creating: build/bin/ 2025-11-03T16:27:30.8138809Z creating: build/bin/CMakeFiles/ 2025-11-03T16:27:30.8139099Z inflating: build/bin/cmake_install.cmake 2025-11-03T16:27:30.8139326Z inflating: build/bin/CTestTestfile.cmake 2025-11-03T16:27:30.8537812Z inflating: build/bin/protoc-3.13.0.0 2025-11-03T16:27:30.8936061Z inflating: build/bin/protoc 2025-11-03T16:27:30.8984879Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-11-03T16:27:30.9036748Z inflating: build/bin/c10_AllocatorConfig_test 2025-11-03T16:27:30.9087292Z inflating: build/bin/c10_Device_test 2025-11-03T16:27:30.9144902Z inflating: build/bin/c10_DispatchKeySet_test 2025-11-03T16:27:30.9194218Z inflating: build/bin/c10_DeviceGuard_test 2025-11-03T16:27:30.9246431Z inflating: build/bin/c10_Scalar_test 2025-11-03T16:27:30.9294479Z inflating: build/bin/c10_StreamGuard_test 2025-11-03T16:27:30.9349153Z inflating: build/bin/c10_SymInt_test 2025-11-03T16:27:30.9402945Z inflating: build/bin/c10_SizesAndStrides_test 2025-11-03T16:27:30.9456497Z inflating: build/bin/c10_InlineStreamGuard_test 2025-11-03T16:27:30.9509768Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-11-03T16:27:30.9577210Z inflating: build/bin/c10_cow_test 2025-11-03T16:27:30.9625027Z inflating: build/bin/c10_ArrayRef_test 2025-11-03T16:27:30.9672234Z inflating: build/bin/c10_ConstexprCrc_test 2025-11-03T16:27:30.9723841Z inflating: build/bin/c10_Bitset_test 2025-11-03T16:27:30.9778526Z inflating: build/bin/c10_Enumerate_test 2025-11-03T16:27:30.9826796Z inflating: build/bin/c10_DeadlockDetection_test 2025-11-03T16:27:30.9879882Z inflating: build/bin/c10_LeftRight_test 2025-11-03T16:27:30.9930907Z inflating: build/bin/c10_IntrusiveList_test 2025-11-03T16:27:30.9979299Z inflating: build/bin/c10_Half_test 2025-11-03T16:27:31.0032488Z inflating: build/bin/c10_Metaprogramming_test 2025-11-03T16:27:31.0080490Z inflating: build/bin/c10_Semaphore_test 2025-11-03T16:27:31.0132980Z inflating: build/bin/c10_NetworkFlow_test 2025-11-03T16:27:31.0180858Z inflating: build/bin/c10_Synchronized_test 2025-11-03T16:27:31.0233818Z inflating: build/bin/c10_ThreadLocal_test 2025-11-03T16:27:31.0283861Z inflating: build/bin/c10_TypeList_test 2025-11-03T16:27:31.0333744Z inflating: build/bin/c10_TypeIndex_test 2025-11-03T16:27:31.0383592Z inflating: build/bin/c10_accumulate_test 2025-11-03T16:27:31.0430732Z inflating: build/bin/c10_TypeTraits_test 2025-11-03T16:27:31.0484558Z inflating: build/bin/c10_bfloat16_test 2025-11-03T16:27:31.0539713Z inflating: build/bin/c10_complex_math_test 2025-11-03T16:27:31.0590145Z inflating: build/bin/c10_exception_test 2025-11-03T16:27:31.0638350Z inflating: build/bin/c10_error_test 2025-11-03T16:27:31.0691288Z inflating: build/bin/c10_complex_test 2025-11-03T16:27:31.0739627Z inflating: build/bin/c10_bit_cast_test 2025-11-03T16:27:31.0788873Z inflating: build/bin/c10_flags_test 2025-11-03T16:27:31.0837339Z inflating: build/bin/c10_generic_math_test 2025-11-03T16:27:31.0887522Z inflating: build/bin/c10_irange_test 2025-11-03T16:27:31.0939516Z inflating: build/bin/c10_lazy_test 2025-11-03T16:27:31.1089435Z inflating: build/bin/c10_intrusive_ptr_test 2025-11-03T16:27:31.1144534Z inflating: build/bin/c10_logging_test 2025-11-03T16:27:31.1216417Z inflating: build/bin/c10_optional_test 2025-11-03T16:27:31.1275359Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-11-03T16:27:31.1327515Z inflating: build/bin/c10_registry_test 2025-11-03T16:27:31.1467488Z inflating: build/bin/c10_small_vector_test 2025-11-03T16:27:31.1517061Z inflating: build/bin/c10_ssize_test 2025-11-03T16:27:31.1571592Z inflating: build/bin/c10_string_util_test 2025-11-03T16:27:31.1620612Z inflating: build/bin/c10_tempfile_test 2025-11-03T16:27:31.1663218Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-11-03T16:27:31.1710917Z inflating: build/bin/c10_string_view_test 2025-11-03T16:27:31.1765693Z inflating: build/bin/c10_typeid_test 2025-11-03T16:27:31.2286255Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-11-03T16:27:31.2818442Z inflating: build/bin/vec_test_all_types_AVX512 2025-11-03T16:27:31.3358120Z inflating: build/bin/vec_test_all_types_AVX2 2025-11-03T16:27:31.3431938Z inflating: build/bin/test_aoti_abi_check 2025-11-03T16:27:31.3478018Z inflating: build/bin/test_vec_half_DEFAULT 2025-11-03T16:27:31.3528184Z inflating: build/bin/test_vec_half_AVX512 2025-11-03T16:27:31.3575924Z inflating: build/bin/test_vec_half_AVX2 2025-11-03T16:27:31.3626736Z inflating: build/bin/BackoffTest 2025-11-03T16:27:31.3678108Z inflating: build/bin/FileStoreTest 2025-11-03T16:27:31.3729715Z inflating: build/bin/static_runtime_bench 2025-11-03T16:27:31.3955278Z inflating: build/bin/static_runtime_test 2025-11-03T16:27:31.4026693Z inflating: build/bin/Dict_test 2025-11-03T16:27:31.4076724Z inflating: build/bin/Dimname_test 2025-11-03T16:27:31.4139519Z inflating: build/bin/MaybeOwned_test 2025-11-03T16:27:31.4192969Z inflating: build/bin/NamedTensor_test 2025-11-03T16:27:31.4249798Z inflating: build/bin/apply_utils_test 2025-11-03T16:27:31.4305352Z inflating: build/bin/atest 2025-11-03T16:27:31.4368564Z inflating: build/bin/basic 2025-11-03T16:27:31.4421695Z inflating: build/bin/broadcast_test 2025-11-03T16:27:31.4470566Z inflating: build/bin/cpu_allocator_test 2025-11-03T16:27:31.4526494Z inflating: build/bin/cpu_generator_test 2025-11-03T16:27:31.4578143Z inflating: build/bin/cpu_profiling_allocator_test 2025-11-03T16:27:31.4663497Z inflating: build/bin/cpu_rng_test 2025-11-03T16:27:31.4712945Z inflating: build/bin/dlconvertor_test 2025-11-03T16:27:31.4768078Z inflating: build/bin/extension_backend_test 2025-11-03T16:27:31.4821596Z inflating: build/bin/half_test 2025-11-03T16:27:31.4909702Z inflating: build/bin/ivalue_test 2025-11-03T16:27:31.4959018Z inflating: build/bin/lazy_tensor_test 2025-11-03T16:27:31.5011563Z inflating: build/bin/math_kernel_test 2025-11-03T16:27:31.5063553Z inflating: build/bin/memory_format_test 2025-11-03T16:27:31.5115439Z inflating: build/bin/memory_overlapping_test 2025-11-03T16:27:31.5166802Z inflating: build/bin/mobile_memory_cleanup 2025-11-03T16:27:31.5220937Z inflating: build/bin/native_test 2025-11-03T16:27:31.5270130Z inflating: build/bin/operator_name_test 2025-11-03T16:27:31.5319662Z inflating: build/bin/operators_test 2025-11-03T16:27:31.5369805Z inflating: build/bin/packedtensoraccessor_test 2025-11-03T16:27:31.5434054Z inflating: build/bin/pow_test 2025-11-03T16:27:31.5489224Z inflating: build/bin/quantized_test 2025-11-03T16:27:31.5537377Z inflating: build/bin/reduce_ops_test 2025-11-03T16:27:31.5586758Z inflating: build/bin/reportMemoryUsage_test 2025-11-03T16:27:31.5641213Z inflating: build/bin/scalar_tensor_test 2025-11-03T16:27:31.5698115Z inflating: build/bin/scalar_test 2025-11-03T16:27:31.5747409Z inflating: build/bin/StorageUtils_test 2025-11-03T16:27:31.5798304Z inflating: build/bin/stride_properties_test 2025-11-03T16:27:31.5871491Z inflating: build/bin/tensor_iterator_test 2025-11-03T16:27:31.5925266Z inflating: build/bin/test_parallel 2025-11-03T16:27:31.5973580Z inflating: build/bin/thread_init_test 2025-11-03T16:27:31.6027257Z inflating: build/bin/type_ptr_test 2025-11-03T16:27:31.6082405Z inflating: build/bin/type_test 2025-11-03T16:27:31.6134485Z inflating: build/bin/undefined_tensor_test 2025-11-03T16:27:31.6183019Z inflating: build/bin/verify_api_visibility 2025-11-03T16:27:31.6249561Z inflating: build/bin/legacy_vmap_test 2025-11-03T16:27:31.6298819Z inflating: build/bin/weakref_test 2025-11-03T16:27:31.6348881Z inflating: build/bin/wrapdim_test 2025-11-03T16:27:31.6398462Z inflating: build/bin/xla_tensor_test 2025-11-03T16:27:31.6454957Z inflating: build/bin/IListRef_test 2025-11-03T16:27:31.6552471Z inflating: build/bin/List_test 2025-11-03T16:27:31.6616463Z inflating: build/bin/KernelFunction_test 2025-11-03T16:27:31.6727184Z inflating: build/bin/kernel_function_legacy_test 2025-11-03T16:27:31.6815541Z inflating: build/bin/kernel_function_test 2025-11-03T16:27:31.6930233Z inflating: build/bin/kernel_lambda_legacy_test 2025-11-03T16:27:31.7023495Z inflating: build/bin/kernel_lambda_test 2025-11-03T16:27:31.7080945Z inflating: build/bin/kernel_stackbased_test 2025-11-03T16:27:31.7169158Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-11-03T16:27:31.7218003Z inflating: build/bin/CppSignature_test 2025-11-03T16:27:31.7271212Z inflating: build/bin/backend_fallback_test 2025-11-03T16:27:31.7319562Z inflating: build/bin/op_allowlist_test 2025-11-03T16:27:31.7592065Z inflating: build/bin/op_registration_test 2025-11-03T16:27:31.7655538Z inflating: build/bin/inline_container_test 2025-11-03T16:27:31.7961089Z inflating: build/bin/test_lazy 2025-11-03T16:27:31.8016400Z inflating: build/bin/TCPStoreTest 2025-11-03T16:27:31.8068186Z inflating: build/bin/HashStoreTest 2025-11-03T16:27:31.9046943Z inflating: build/bin/test_jit 2025-11-03T16:27:31.9109714Z inflating: build/bin/ProcessGroupGlooTest 2025-11-03T16:27:31.9111386Z inflating: build/bin/example_allreduce 2025-11-03T16:27:31.9168082Z inflating: build/bin/test_aoti_inference 2025-11-03T16:27:31.9222328Z inflating: build/bin/test_dist_autograd 2025-11-03T16:27:31.9287827Z inflating: build/bin/test_cpp_rpc 2025-11-03T16:27:32.0297279Z inflating: build/bin/test_api 2025-11-03T16:27:32.0297746Z inflating: build/bin/parallel_benchmark 2025-11-03T16:27:32.0302334Z inflating: build/bin/torch_shm_manager 2025-11-03T16:27:32.0302696Z creating: .additional_ci_files/ 2025-11-03T16:27:32.0351985Z inflating: .additional_ci_files/test-times.json 2025-11-03T16:27:32.0535557Z inflating: .additional_ci_files/test-class-times.json 2025-11-03T16:27:32.0707947Z ##[group]Run rm artifacts.zip 2025-11-03T16:27:32.0708161Z rm artifacts.zip 2025-11-03T16:27:32.0712982Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:32.0713405Z env: 2025-11-03T16:27:32.0713567Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:32.0713747Z ##[endgroup] 2025-11-03T16:27:32.1182255Z ##[group]Run df -H 2025-11-03T16:27:32.1182453Z df -H 2025-11-03T16:27:32.1191086Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:32.1191328Z env: 2025-11-03T16:27:32.1191509Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:32.1191684Z ##[endgroup] 2025-11-03T16:27:32.1231146Z Filesystem Size Used Avail Use% Mounted on 2025-11-03T16:27:32.1233239Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-11-03T16:27:32.1233503Z tmpfs 67G 0 67G 0% /dev/shm 2025-11-03T16:27:32.1233779Z tmpfs 27G 791k 27G 1% /run 2025-11-03T16:27:32.1234130Z /dev/nvme0n1p1 215G 71G 144G 34% / 2025-11-03T16:27:32.1234365Z tmpfs 67G 13k 67G 1% /tmp 2025-11-03T16:27:32.1234591Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-11-03T16:27:32.1262172Z Prepare all required actions 2025-11-03T16:27:32.1262938Z Getting action download info 2025-11-03T16:27:32.3405866Z ##[group]Run ./.github/actions/download-td-artifacts 2025-11-03T16:27:32.3406097Z with: 2025-11-03T16:27:32.3406238Z env: 2025-11-03T16:27:32.3406385Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:32.3406547Z ##[endgroup] 2025-11-03T16:27:32.3547523Z ##[group]Run seemethere/download-artifact-s3@v4 2025-11-03T16:27:32.3547739Z with: 2025-11-03T16:27:32.3547876Z name: td_results 2025-11-03T16:27:32.3548042Z s3-bucket: gha-artifacts 2025-11-03T16:27:32.3548216Z region: us-east-1 2025-11-03T16:27:32.3548365Z env: 2025-11-03T16:27:32.3548501Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:32.3548667Z ##[endgroup] 2025-11-03T16:27:33.0414668Z (node:48493) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-11-03T16:27:33.0419049Z 2025-11-03T16:27:33.0421155Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-11-03T16:27:33.0421640Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-11-03T16:27:33.0426365Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-11-03T16:27:33.1224354Z Found 0 objects with prefix pytorch/pytorch/19040285537/td_results/ 2025-11-03T16:27:33.1230599Z Artifact download has finished successfully 2025-11-03T16:27:33.1618383Z ##[group]Run mkdir -p .additional_ci_files 2025-11-03T16:27:33.1618632Z mkdir -p .additional_ci_files 2025-11-03T16:27:33.1618895Z mv td_results.json .additional_ci_files/td_results.json || true 2025-11-03T16:27:33.1624382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:33.1624607Z env: 2025-11-03T16:27:33.1624759Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:33.1624927Z ##[endgroup] 2025-11-03T16:27:33.1674786Z mv: cannot stat 'td_results.json': No such file or directory 2025-11-03T16:27:33.6409767Z ##[group]Run .github/scripts/parse_ref.py 2025-11-03T16:27:33.6410036Z .github/scripts/parse_ref.py 2025-11-03T16:27:33.6415305Z shell: /usr/bin/bash -e {0} 2025-11-03T16:27:33.6415482Z env: 2025-11-03T16:27:33.6415627Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:33.6415793Z ##[endgroup] 2025-11-03T16:27:33.7281696Z Setting output branch=main 2025-11-03T16:27:33.7369853Z Prepare all required actions 2025-11-03T16:27:33.7370152Z Getting action download info 2025-11-03T16:27:33.8829223Z ##[group]Run ./.github/actions/filter-test-configs 2025-11-03T16:27:33.8829484Z with: 2025-11-03T16:27:33.8829926Z github-token: *** 2025-11-03T16:27:33.8831569Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-11-03T16:27:33.8833349Z job-name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:33.8833685Z env: 2025-11-03T16:27:33.8833852Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:33.8834138Z ##[endgroup] 2025-11-03T16:27:33.8896694Z ##[group]Run nick-fields/retry@v3.0.0 2025-11-03T16:27:33.8896898Z with: 2025-11-03T16:27:33.8897036Z shell: bash 2025-11-03T16:27:33.8897204Z timeout_minutes: 10 2025-11-03T16:27:33.8897359Z max_attempts: 5 2025-11-03T16:27:33.8897513Z retry_wait_seconds: 30 2025-11-03T16:27:33.8897957Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-11-03T16:27:33.8898409Z polling_interval_seconds: 1 2025-11-03T16:27:33.8898596Z warning_on_retry: true 2025-11-03T16:27:33.8898765Z continue_on_error: false 2025-11-03T16:27:33.8898929Z env: 2025-11-03T16:27:33.8899061Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:33.8899360Z GITHUB_TOKEN: *** 2025-11-03T16:27:33.8899514Z ##[endgroup] 2025-11-03T16:27:34.0118563Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-11-03T16:27:34.1779970Z Defaulting to user installation because normal site-packages is not writeable 2025-11-03T16:27:34.3211880Z Collecting requests==2.27.1 2025-11-03T16:27:34.3347032Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-11-03T16:27:34.5768918Z Collecting pyyaml==6.0.2 2025-11-03T16:27:34.5799926Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-11-03T16:27:34.6827290Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-11-03T16:27:34.7842842Z Collecting certifi>=2017.4.17 2025-11-03T16:27:34.7873620Z Downloading certifi-2025.10.5-py3-none-any.whl (163 kB) 2025-11-03T16:27:35.1097247Z Collecting charset-normalizer~=2.0.0 2025-11-03T16:27:35.1132255Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-11-03T16:27:35.1174645Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-11-03T16:27:35.1738165Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-11-03T16:27:35.2645631Z Successfully installed certifi-2025.10.5 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-11-03T16:27:35.9526489Z Command completed after 1 attempt(s). 2025-11-03T16:27:35.9587433Z ##[group]Run set -x 2025-11-03T16:27:35.9587619Z set -x 2025-11-03T16:27:35.9587772Z  2025-11-03T16:27:35.9588013Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-11-03T16:27:35.9588301Z # in runner workspace 2025-11-03T16:27:35.9588677Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-11-03T16:27:35.9594099Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:35.9594350Z env: 2025-11-03T16:27:35.9594504Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:35.9594686Z ##[endgroup] 2025-11-03T16:27:35.9620457Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-11-03T16:27:35.9759019Z Setting output branch=main 2025-11-03T16:27:35.9808701Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-11-03T16:27:35.9808974Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-11-03T16:27:35.9809187Z echo "Job name: ${JOB_NAME}" 2025-11-03T16:27:35.9809374Z  2025-11-03T16:27:35.9809610Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-11-03T16:27:35.9809881Z # in runner workspace 2025-11-03T16:27:35.9810159Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-11-03T16:27:35.9810443Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-11-03T16:27:35.9810651Z  --job-name "${JOB_NAME}" \ 2025-11-03T16:27:35.9812131Z  --test-matrix "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" \ 2025-11-03T16:27:35.9813847Z  --selected-test-configs "" \ 2025-11-03T16:27:35.9814063Z  --pr-number "${PR_NUMBER}" \ 2025-11-03T16:27:35.9814262Z  --tag "${TAG}" \ 2025-11-03T16:27:35.9814443Z  --event-name "${EVENT_NAME}" \ 2025-11-03T16:27:35.9814643Z  --schedule "${SCHEDULE}" \ 2025-11-03T16:27:35.9814838Z  --branch "${HEAD_BRANCH}" 2025-11-03T16:27:35.9819010Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:35.9819349Z env: 2025-11-03T16:27:35.9819491Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:35.9819936Z GITHUB_TOKEN: *** 2025-11-03T16:27:35.9820225Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:35.9820518Z PR_NUMBER: 2025-11-03T16:27:35.9820658Z TAG: 2025-11-03T16:27:35.9820799Z EVENT_NAME: push 2025-11-03T16:27:35.9820953Z SCHEDULE: 2025-11-03T16:27:35.9821099Z HEAD_BRANCH: main 2025-11-03T16:27:35.9821252Z ##[endgroup] 2025-11-03T16:27:35.9844362Z Workflow: inductor 2025-11-03T16:27:35.9846461Z Job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:36.1514822Z Setting output keep-going=True 2025-11-03T16:27:36.1519210Z Setting output ci-verbose-test-logs=False 2025-11-03T16:27:36.1521333Z Setting output ci-test-showlocals=False 2025-11-03T16:27:36.1526161Z Setting output ci-no-test-timeout=False 2025-11-03T16:27:36.1528245Z Setting output ci-no-td=False 2025-11-03T16:27:36.1533429Z Setting output ci-td-distributed=False 2025-11-03T16:27:36.1538043Z Setting output is-unstable=False 2025-11-03T16:27:36.1542543Z Setting output reenabled-issues= 2025-11-03T16:27:36.1546046Z Setting output test-matrix={"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-11-03T16:27:36.1547592Z Setting output is-test-matrix-empty=False 2025-11-03T16:27:36.1645021Z ##[group]Run echo "Filtered matrix:" 2025-11-03T16:27:36.1645258Z echo "Filtered matrix:" 2025-11-03T16:27:36.1646802Z echo "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" 2025-11-03T16:27:36.1648309Z  2025-11-03T16:27:36.1648451Z echo 2025-11-03T16:27:36.1648629Z echo "Is the current job unstable? False" 2025-11-03T16:27:36.1648855Z  2025-11-03T16:27:36.1648987Z echo 2025-11-03T16:27:36.1649151Z echo "Is keep-going label set? True" 2025-11-03T16:27:36.1649340Z  2025-11-03T16:27:36.1649471Z echo 2025-11-03T16:27:36.1649623Z echo "Reenabled issues? " 2025-11-03T16:27:36.1654216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:36.1654439Z env: 2025-11-03T16:27:36.1654590Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:36.1654760Z ##[endgroup] 2025-11-03T16:27:36.1675630Z Filtered matrix: 2025-11-03T16:27:36.1678991Z {include: [{config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: linux.24xl.spr-metal}]} 2025-11-03T16:27:36.1680597Z 2025-11-03T16:27:36.1680680Z Is the current job unstable? False 2025-11-03T16:27:36.1680829Z 2025-11-03T16:27:36.1680908Z Is keep-going label set? True 2025-11-03T16:27:36.1681031Z 2025-11-03T16:27:36.1681107Z Reenabled issues? 2025-11-03T16:27:36.1715167Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-11-03T16:27:36.1715514Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-11-03T16:27:36.1719221Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:36.1719447Z env: 2025-11-03T16:27:36.1719592Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:36.1719759Z JOB_TIMEOUT: 240 2025-11-03T16:27:36.1719906Z ##[endgroup] 2025-11-03T16:27:36.1767555Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-11-03T16:27:36.1767887Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-11-03T16:27:36.1768163Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-11-03T16:27:36.1771741Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:27:36.1771976Z env: 2025-11-03T16:27:36.1772130Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:36.1772304Z ##[endgroup] 2025-11-03T16:27:36.1866197Z ##[group]Run set -x 2025-11-03T16:27:36.1866443Z set -x 2025-11-03T16:27:36.1866592Z  2025-11-03T16:27:36.1866777Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-11-03T16:27:36.1867037Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-11-03T16:27:36.1867284Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-11-03T16:27:36.1867502Z  TEST_COMMAND=.ci/onnx/test.sh 2025-11-03T16:27:36.1867691Z else 2025-11-03T16:27:36.1867863Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-11-03T16:27:36.1868053Z fi 2025-11-03T16:27:36.1868184Z  2025-11-03T16:27:36.1868357Z # Leaving 1GB for the runner and other things 2025-11-03T16:27:36.1868700Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-11-03T16:27:36.1869212Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-11-03T16:27:36.1869612Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-11-03T16:27:36.1869929Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-11-03T16:27:36.1870180Z  2025-11-03T16:27:36.1870358Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-11-03T16:27:36.1870563Z  SHM_OPTS= 2025-11-03T16:27:36.1870730Z  JENKINS_USER= 2025-11-03T16:27:36.1870954Z  # ensure that docker container cleanly exits in 12 hours 2025-11-03T16:27:36.1871239Z  # if for some reason cleanup action doesn't stop container 2025-11-03T16:27:36.1871476Z  # when job is cancelled 2025-11-03T16:27:36.1871666Z  DOCKER_SHELL_CMD="sleep 12h" 2025-11-03T16:27:36.1871854Z else 2025-11-03T16:27:36.1872022Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-11-03T16:27:36.1872233Z  JENKINS_USER="--user jenkins" 2025-11-03T16:27:36.1872424Z  DOCKER_SHELL_CMD= 2025-11-03T16:27:36.1872592Z fi 2025-11-03T16:27:36.1872731Z  2025-11-03T16:27:36.1872943Z # detached container should get cleaned up by teardown_ec2_linux 2025-11-03T16:27:36.1873331Z # TODO: Stop building test binaries as part of the build phase 2025-11-03T16:27:36.1873681Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-11-03T16:27:36.1874130Z # shellcheck disable=SC2086,SC2090 2025-11-03T16:27:36.1874360Z container_name=$(docker run \ 2025-11-03T16:27:36.1874567Z  ${GPU_FLAG:-} \ 2025-11-03T16:27:36.1874763Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-11-03T16:27:36.1874985Z  -e BUILD_ENVIRONMENT \ 2025-11-03T16:27:36.1875186Z  -e PR_NUMBER \ 2025-11-03T16:27:36.1875370Z  -e GITHUB_ACTIONS \ 2025-11-03T16:27:36.1875565Z  -e GITHUB_REPOSITORY \ 2025-11-03T16:27:36.1875763Z  -e GITHUB_WORKFLOW \ 2025-11-03T16:27:36.1875951Z  -e GITHUB_JOB \ 2025-11-03T16:27:36.1876127Z  -e GITHUB_RUN_ID \ 2025-11-03T16:27:36.1876311Z  -e GITHUB_RUN_NUMBER \ 2025-11-03T16:27:36.1876500Z  -e GITHUB_RUN_ATTEMPT \ 2025-11-03T16:27:36.1876685Z  -e JOB_ID \ 2025-11-03T16:27:36.1876849Z  -e JOB_NAME \ 2025-11-03T16:27:36.1877010Z  -e BASE_SHA \ 2025-11-03T16:27:36.1877173Z  -e BRANCH \ 2025-11-03T16:27:36.1877332Z  -e SHA1 \ 2025-11-03T16:27:36.1877499Z  -e AWS_DEFAULT_REGION \ 2025-11-03T16:27:36.1877678Z  -e IN_WHEEL_TEST \ 2025-11-03T16:27:36.1877856Z  -e SHARD_NUMBER \ 2025-11-03T16:27:36.1878030Z  -e TEST_CONFIG \ 2025-11-03T16:27:36.1878207Z  -e NUM_TEST_SHARDS \ 2025-11-03T16:27:36.1878385Z  -e REENABLED_ISSUES \ 2025-11-03T16:27:36.1878577Z  -e CONTINUE_THROUGH_ERROR \ 2025-11-03T16:27:36.1878841Z  -e VERBOSE_TEST_LOGS \ 2025-11-03T16:27:36.1879032Z  -e TEST_SHOWLOCALS \ 2025-11-03T16:27:36.1879204Z  -e NO_TEST_TIMEOUT \ 2025-11-03T16:27:36.1879378Z  -e NO_TD \ 2025-11-03T16:27:36.1879543Z  -e TD_DISTRIBUTED \ 2025-11-03T16:27:36.1879723Z  -e PR_LABELS \ 2025-11-03T16:27:36.1879907Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-11-03T16:27:36.1880114Z  -e SCCACHE_BUCKET \ 2025-11-03T16:27:36.1880292Z  -e SCCACHE_REGION \ 2025-11-03T16:27:36.1880468Z  -e XLA_CUDA \ 2025-11-03T16:27:36.1880654Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-11-03T16:27:36.1880872Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-11-03T16:27:36.1881101Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-11-03T16:27:36.1881327Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-11-03T16:27:36.1881536Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-11-03T16:27:36.1881735Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-11-03T16:27:36.1881946Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-11-03T16:27:36.1882144Z  -e DASHBOARD_TAG \ 2025-11-03T16:27:36.1882334Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-11-03T16:27:36.1882555Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-11-03T16:27:36.1882815Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-11-03T16:27:36.1883084Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-11-03T16:27:36.1883327Z  --security-opt seccomp=unconfined \ 2025-11-03T16:27:36.1883538Z  --cap-add=SYS_PTRACE \ 2025-11-03T16:27:36.1883719Z  --ipc=host \ 2025-11-03T16:27:36.1883889Z  ${SHM_OPTS} \ 2025-11-03T16:27:36.1884052Z  --tty \ 2025-11-03T16:27:36.1884211Z  --detach \ 2025-11-03T16:27:36.1884381Z  --name="${container_name}" \ 2025-11-03T16:27:36.1884573Z  ${JENKINS_USER} \ 2025-11-03T16:27:36.1884800Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-11-03T16:27:36.1885039Z  -w /var/lib/jenkins/workspace \ 2025-11-03T16:27:36.1885275Z  "${DOCKER_IMAGE}" \ 2025-11-03T16:27:36.1885456Z  ${DOCKER_SHELL_CMD} 2025-11-03T16:27:36.1885627Z ) 2025-11-03T16:27:36.1885842Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-11-03T16:27:36.1886078Z  2025-11-03T16:27:36.1886261Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-11-03T16:27:36.1886612Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-11-03T16:27:36.1886916Z fi 2025-11-03T16:27:36.1887056Z  2025-11-03T16:27:36.1887344Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-11-03T16:27:36.1891076Z shell: /usr/bin/bash -e {0} 2025-11-03T16:27:36.1891250Z env: 2025-11-03T16:27:36.1891403Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:27:36.1891608Z BUILD_ENVIRONMENT: linux-jammy-py3.10-gcc11-build 2025-11-03T16:27:36.1891830Z PR_NUMBER: 2025-11-03T16:27:36.1891995Z GITHUB_REPOSITORY: pytorch/pytorch 2025-11-03T16:27:36.1892188Z GITHUB_WORKFLOW: inductor 2025-11-03T16:27:36.1892349Z GITHUB_JOB: test 2025-11-03T16:27:36.1892506Z GITHUB_RUN_ID: 19040285537 2025-11-03T16:27:36.1892680Z GITHUB_RUN_NUMBER: 163246 2025-11-03T16:27:36.1892846Z GITHUB_RUN_ATTEMPT: 1 2025-11-03T16:27:36.1892998Z JOB_ID: 54378387707 2025-11-03T16:27:36.1893277Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:36.1893570Z BRANCH: main 2025-11-03T16:27:36.1893746Z SHA1: 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:36.1893972Z BASE_SHA: 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:36.1894203Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-11-03T16:27:36.1894402Z SHARD_NUMBER: 1 2025-11-03T16:27:36.1894620Z NUM_TEST_SHARDS: 1 2025-11-03T16:27:36.1894773Z EXTRA_FLAGS: 2025-11-03T16:27:36.1894918Z OP_BENCHMARK_TESTS: 2025-11-03T16:27:36.1895079Z REENABLED_ISSUES: 2025-11-03T16:27:36.1895243Z CONTINUE_THROUGH_ERROR: True 2025-11-03T16:27:36.1895419Z VERBOSE_TEST_LOGS: False 2025-11-03T16:27:36.1895581Z TEST_SHOWLOCALS: False 2025-11-03T16:27:36.1895745Z NO_TEST_TIMEOUT: False 2025-11-03T16:27:36.1895903Z NO_TD: False 2025-11-03T16:27:36.1896043Z TD_DISTRIBUTED: False 2025-11-03T16:27:36.1896247Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-11-03T16:27:36.1896471Z SCCACHE_REGION: us-east-1 2025-11-03T16:27:36.1896638Z SHM_SIZE: 1g 2025-11-03T16:27:36.1897113Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:27:36.1897598Z XLA_CUDA: 2025-11-03T16:27:36.1897826Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-11-03T16:27:36.1898103Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-11-03T16:27:36.1898307Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-11-03T16:27:36.1898486Z DASHBOARD_TAG: 2025-11-03T16:27:36.1898807Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-11-03T16:27:36.1899068Z HUGGING_FACE_HUB_TOKEN: *** 2025-11-03T16:27:36.1899325Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-11-03T16:27:36.1899640Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:27:36.1899945Z ##[endgroup] 2025-11-03T16:27:36.1922721Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-11-03T16:27:36.1924436Z + [[ linux-jammy-py3.10-gcc11-build == *onnx* ]] 2025-11-03T16:27:36.1924798Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-11-03T16:27:36.1929258Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-11-03T16:27:36.1941601Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-11-03T16:27:36.1943429Z + TOTAL_MEMORY_WITH_SWAP=125 2025-11-03T16:27:36.1943830Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-11-03T16:27:36.1948612Z + SHM_OPTS=--shm-size=1g 2025-11-03T16:27:36.1948989Z + JENKINS_USER='--user jenkins' 2025-11-03T16:27:36.1955554Z + DOCKER_SHELL_CMD= 2025-11-03T16:27:36.1958865Z +++ nproc --ignore=2 2025-11-03T16:27:36.1978887Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_19040285537 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:27:46.6306814Z + container_name=06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:27:46.6307513Z + echo DOCKER_CONTAINER_ID=06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:27:46.6307888Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-11-03T16:27:46.6312154Z ++ echo dist/torch-2.10.0a0+git3f6538f-cp310-cp310-linux_x86_64.whl 2025-11-03T16:27:46.6314236Z + docker exec -t 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a sh -c 'python3 -m pip install dist/torch-2.10.0a0+git3f6538f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-11-03T16:27:47.0478861Z Processing ./dist/torch-2.10.0a0+git3f6538f-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+git3f6538f) 2025-11-03T16:27:47.2563168Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (3.18.0) 2025-11-03T16:27:47.2565332Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (4.15.0) 2025-11-03T16:27:47.2566136Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (1.13.3) 2025-11-03T16:27:47.2571485Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (2.8.8) 2025-11-03T16:27:47.2573619Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (3.1.6) 2025-11-03T16:27:47.2577922Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (2025.9.0) 2025-11-03T16:27:47.2585625Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (3.3.0) 2025-11-03T16:27:47.2853896Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (1.22.4) 2025-11-03T16:27:47.2867129Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (1.3.0) 2025-11-03T16:27:47.2911526Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+git3f6538f->torch==2.10.0a0+git3f6538f) (3.0.3) 2025-11-03T16:27:48.0033459Z Installing collected packages: torch 2025-11-03T16:27:54.8408477Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-11-03T16:27:54.8409105Z dall-e 0.1 requires torchvision, which is not installed. 2025-11-03T16:27:54.8409408Z effdet 0.4.1 requires torchvision, which is not installed. 2025-11-03T16:27:54.8409727Z python-doctr 1.0.0 requires torchvision>=0.15.0, which is not installed. 2025-11-03T16:27:54.8410117Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-11-03T16:27:54.8410626Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-11-03T16:27:54.8411106Z timm 1.0.14 requires torchvision, which is not installed. 2025-11-03T16:27:54.8411459Z Successfully installed torch-2.10.0a0+git3f6538f 2025-11-03T16:27:54.9552332Z + export TERM=vt100 2025-11-03T16:27:54.9554155Z + TERM=vt100 2025-11-03T16:27:54.9554403Z ++ dirname .ci/pytorch/test.sh 2025-11-03T16:27:54.9567768Z + source .ci/pytorch/common.sh 2025-11-03T16:27:54.9568023Z +++ dirname .ci/pytorch/common.sh 2025-11-03T16:27:54.9572513Z ++ source .ci/pytorch/common_utils.sh 2025-11-03T16:27:54.9572954Z +++ declare -f -t trap_add 2025-11-03T16:27:54.9579064Z ++ set -ex -o pipefail 2025-11-03T16:27:54.9579332Z ++ [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-11-03T16:27:54.9579568Z ++ BUILD_TEST_LIBTORCH=0 2025-11-03T16:27:54.9579763Z ++ dirname .ci/pytorch/test.sh 2025-11-03T16:27:54.9587556Z + source .ci/pytorch/common-build.sh 2025-11-03T16:27:54.9588304Z ++ [[ linux-jammy-py3.10-gcc11-build != *win-* ]] 2025-11-03T16:27:54.9601673Z ++++ dirname .ci/pytorch/common-build.sh 2025-11-03T16:27:54.9613369Z +++ cd .ci/pytorch 2025-11-03T16:27:54.9617889Z +++ pwd -P 2025-11-03T16:27:54.9621355Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-11-03T16:27:54.9623443Z ++ [[ linux-jammy-py3.10-gcc11-build == *-pch* ]] 2025-11-03T16:27:54.9623801Z ++ which sccache 2025-11-03T16:27:54.9630734Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-11-03T16:27:54.9631018Z ++ sccache --stop-server 2025-11-03T16:27:54.9659988Z ++ true 2025-11-03T16:27:54.9662277Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-11-03T16:27:54.9671401Z ++ trap_add sccache_epilogue EXIT 2025-11-03T16:27:54.9676400Z ++ trap_add_cmd=sccache_epilogue 2025-11-03T16:27:54.9678279Z ++ shift 2025-11-03T16:27:54.9678581Z ++ for trap_add_name in "$@" 2025-11-03T16:27:54.9684167Z ++++ trap -p EXIT 2025-11-03T16:27:54.9689564Z +++ eval 'extract_trap_cmd ' 2025-11-03T16:27:54.9689958Z ++++ extract_trap_cmd 2025-11-03T16:27:54.9690239Z ++++ printf '%s\n' '' 2025-11-03T16:27:54.9690458Z +++ printf '%s\n' sccache_epilogue 2025-11-03T16:27:54.9690718Z ++ trap -- ' 2025-11-03T16:27:54.9695745Z sccache_epilogue' EXIT 2025-11-03T16:27:54.9700500Z ++ [[ -n 1 ]] 2025-11-03T16:27:54.9704591Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-11-03T16:27:54.9709458Z Skipping sccache server initialization, setting environment variables 2025-11-03T16:27:54.9709790Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-11-03T16:27:54.9709990Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-11-03T16:27:54.9710235Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-11-03T16:27:54.9710524Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-11-03T16:27:54.9719915Z ++ export RUST_LOG=sccache::server=error 2025-11-03T16:27:54.9720175Z ++ RUST_LOG=sccache::server=error 2025-11-03T16:27:54.9720377Z ++ sccache --zero-stats 2025-11-03T16:27:55.1111100Z Statistics zeroed. 2025-11-03T16:27:55.1111516Z ++ which ccache 2025-11-03T16:27:55.1142615Z + [[ linux-jammy-py3.10-gcc11-build != *rocm* ]] 2025-11-03T16:27:55.1142973Z + [[ linux-jammy-py3.10-gcc11-build != *s390x* ]] 2025-11-03T16:27:55.1147420Z + [[ -d /var/lib/jenkins/workspace ]] 2025-11-03T16:27:55.1147808Z ++ stat -c %u /var/lib/jenkins/workspace 2025-11-03T16:27:55.1155777Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-11-03T16:27:55.1157659Z + trap_add cleanup_workspace EXIT 2025-11-03T16:27:55.1158014Z + trap_add_cmd=cleanup_workspace 2025-11-03T16:27:55.1163555Z + shift 2025-11-03T16:27:55.1165506Z + for trap_add_name in "$@" 2025-11-03T16:27:55.1165903Z +++ trap -p EXIT 2025-11-03T16:27:55.1169171Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-11-03T16:27:55.1169441Z sccache_epilogue'\'' EXIT' 2025-11-03T16:27:55.1169661Z +++ extract_trap_cmd trap -- ' 2025-11-03T16:27:55.1169852Z sccache_epilogue' EXIT 2025-11-03T16:27:55.1170031Z +++ printf '%s\n' ' 2025-11-03T16:27:55.1170196Z sccache_epilogue' 2025-11-03T16:27:55.1170372Z ++ printf '%s\n' cleanup_workspace 2025-11-03T16:27:55.1170582Z + trap -- ' 2025-11-03T16:27:55.1170736Z sccache_epilogue 2025-11-03T16:27:55.1170902Z cleanup_workspace' EXIT 2025-11-03T16:27:55.1171117Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-11-03T16:27:55.5288229Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-11-03T16:27:55.5308362Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-11-03T16:27:55.5313059Z + echo 'Environment variables:' 2025-11-03T16:27:55.5315071Z Environment variables: 2025-11-03T16:27:55.5315391Z + env 2025-11-03T16:27:55.5321095Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-11-03T16:27:55.5325550Z CONTINUE_THROUGH_ERROR=True 2025-11-03T16:27:55.5327784Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-11-03T16:27:55.5328386Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-11-03T16:27:55.5332792Z HOSTNAME=06ea20fe7f0f 2025-11-03T16:27:55.5337588Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5339722Z GITHUB_ACTION=__run_3 2025-11-03T16:27:55.5344047Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-11-03T16:27:55.5346758Z GITHUB_RUN_NUMBER=163246 2025-11-03T16:27:55.5347130Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-11-03T16:27:55.5353075Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-11-03T16:27:55.5357608Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-11-03T16:27:55.5362299Z SCCACHE_IDLE_TIMEOUT=0 2025-11-03T16:27:55.5364337Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-11-03T16:27:55.5364682Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-11-03T16:27:55.5369255Z GITHUB_REF_TYPE=branch 2025-11-03T16:27:55.5371648Z BASE_SHA=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5376017Z XLA_CUDA= 2025-11-03T16:27:55.5379892Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-11-03T16:27:55.5380342Z HUGGING_FACE_HUB_TOKEN=*** 2025-11-03T16:27:55.5380774Z *** 2025-11-03T16:27:55.5380930Z GITHUB_REPOSITORY_ID=65600975 2025-11-03T16:27:55.5381123Z GITHUB_ACTIONS=true 2025-11-03T16:27:55.5381355Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-11-03T16:27:55.5381624Z SHA1=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5381891Z GITHUB_SHA=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5382235Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-11-03T16:27:55.5382534Z UCC_HOME=/usr 2025-11-03T16:27:55.5382695Z VERBOSE_TEST_LOGS=False 2025-11-03T16:27:55.5382871Z GITHUB_REF=refs/heads/main 2025-11-03T16:27:55.5383049Z SHARD_NUMBER=1 2025-11-03T16:27:55.5383215Z GITHUB_REF_PROTECTED=true 2025-11-03T16:27:55.5383395Z HOME=/var/lib/jenkins 2025-11-03T16:27:55.5383588Z GITHUB_API_URL=https://api.github.com 2025-11-03T16:27:55.5383817Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-11-03T16:27:55.5384005Z UCX_COMMIT= 2025-11-03T16:27:55.5384149Z USE_SYSTEM_NCCL=1 2025-11-03T16:27:55.5384296Z NUM_TEST_SHARDS=1 2025-11-03T16:27:55.5384449Z UCX_HOME=/usr 2025-11-03T16:27:55.5384805Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5385303Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:55.5386062Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5386528Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-11-03T16:27:55.5386825Z GITHUB_EVENT_NAME=push 2025-11-03T16:27:55.5386998Z DASHBOARD_TAG= 2025-11-03T16:27:55.5387154Z GITHUB_RUN_ID=19040285537 2025-11-03T16:27:55.5387328Z INSTALLED_OPENBLAS= 2025-11-03T16:27:55.5387689Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5388087Z GITHUB_ACTOR=pytorchmergebot 2025-11-03T16:27:55.5388259Z PR_NUMBER= 2025-11-03T16:27:55.5388405Z DESIRED_CUDA= 2025-11-03T16:27:55.5388559Z GITHUB_RUN_ATTEMPT=1 2025-11-03T16:27:55.5388736Z ANACONDA_PYTHON_VERSION=3.10 2025-11-03T16:27:55.5388946Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-11-03T16:27:55.5389160Z TERM=vt100 2025-11-03T16:27:55.5389310Z INSTALLED_VISION=yes 2025-11-03T16:27:55.5389470Z BRANCH=main 2025-11-03T16:27:55.5389619Z SCCACHE_REGION=us-east-1 2025-11-03T16:27:55.5389801Z OPENSSL_ROOT_DIR=/opt/openssl 2025-11-03T16:27:55.5389989Z BUILD_AOT_INDUCTOR_TEST=yes 2025-11-03T16:27:55.5390170Z CUDA_PATH=/usr/local/cuda 2025-11-03T16:27:55.5390478Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-11-03T16:27:55.5390818Z GITHUB_SERVER_URL=https://github.com 2025-11-03T16:27:55.5391012Z UCC_COMMIT= 2025-11-03T16:27:55.5391155Z REENABLED_ISSUES= 2025-11-03T16:27:55.5391296Z DOCS=yes 2025-11-03T16:27:55.5391435Z SHLVL=1 2025-11-03T16:27:55.5391573Z MAX_JOBS=30 2025-11-03T16:27:55.5391712Z GITHUB_ACTOR_ID=97764156 2025-11-03T16:27:55.5391932Z GITHUB_WORKFLOW_SHA=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5392221Z GITHUB_REF_NAME=main 2025-11-03T16:27:55.5392474Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-11-03T16:27:55.5392738Z GITHUB_JOB=test 2025-11-03T16:27:55.5392888Z NO_TEST_TIMEOUT=False 2025-11-03T16:27:55.5393054Z TD_DISTRIBUTED=False 2025-11-03T16:27:55.5393232Z GITHUB_REPOSITORY=pytorch/pytorch 2025-11-03T16:27:55.5393422Z GITHUB_RETENTION_DAYS=90 2025-11-03T16:27:55.5393597Z OPENSSL_DIR=/opt/openssl 2025-11-03T16:27:55.5393776Z GITHUB_ACTION_REPOSITORY= 2025-11-03T16:27:55.5394341Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-11-03T16:27:55.5394835Z GITHUB_BASE_REF= 2025-11-03T16:27:55.5394994Z INSTALLED_ACL= 2025-11-03T16:27:55.5395298Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:27:55.5395625Z CI=true 2025-11-03T16:27:55.5395787Z GITHUB_REPOSITORY_OWNER=pytorch 2025-11-03T16:27:55.5396034Z RUST_LOG=sccache::server=error 2025-11-03T16:27:55.5396232Z JOB_ID=54378387707 2025-11-03T16:27:55.5396389Z GITHUB_HEAD_REF= 2025-11-03T16:27:55.5396552Z GITHUB_ACTION_REF= 2025-11-03T16:27:55.5396743Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-11-03T16:27:55.5396971Z TEST_SHOWLOCALS=False 2025-11-03T16:27:55.5397140Z GITHUB_WORKFLOW=inductor 2025-11-03T16:27:55.5397318Z DEBIAN_FRONTEND=noninteractive 2025-11-03T16:27:55.5397696Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5398063Z NO_TD=False 2025-11-03T16:27:55.5398233Z SKIP_SCCACHE_INITIALIZATION=1 2025-11-03T16:27:55.5398423Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-11-03T16:27:55.5398602Z _=/usr/bin/env 2025-11-03T16:27:55.5398801Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-11-03T16:27:55.5552817Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-11-03T16:27:55.5553240Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-11-03T16:27:55.5553583Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-11-03T16:27:55.5554216Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-11-03T16:27:55.5554490Z + BUILD_DIR=build 2025-11-03T16:27:55.5554672Z + BUILD_RENAMED_DIR=build_renamed 2025-11-03T16:27:55.5554880Z + BUILD_BIN_DIR=build/bin 2025-11-03T16:27:55.5555053Z + SHARD_NUMBER=1 2025-11-03T16:27:55.5555216Z + NUM_TEST_SHARDS=1 2025-11-03T16:27:55.5555403Z + export TORCH_SERIALIZATION_DEBUG=1 2025-11-03T16:27:55.5555602Z + TORCH_SERIALIZATION_DEBUG=1 2025-11-03T16:27:55.5555774Z + export VALGRIND=ON 2025-11-03T16:27:55.5555933Z + VALGRIND=ON 2025-11-03T16:27:55.5556129Z + [[ linux-jammy-py3.10-gcc11-build == *clang9* ]] 2025-11-03T16:27:55.5556365Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-11-03T16:27:55.5556559Z + detect_cuda_arch 2025-11-03T16:27:55.5556743Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-11-03T16:27:55.5556977Z + [[ linux-jammy-py3.10-gcc11-build == *s390x* ]] 2025-11-03T16:27:55.5557175Z + [[ 0 == \1 ]] 2025-11-03T16:27:55.5557312Z + [[ True == \1 ]] 2025-11-03T16:27:55.5557487Z + [[ linux-jammy-py3.10-gcc11-build != *bazel* ]] 2025-11-03T16:27:55.5557816Z ++ realpath build/custom_test_artifacts 2025-11-03T16:27:55.5566879Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-11-03T16:27:55.5571263Z + [[ -n '' ]] 2025-11-03T16:27:55.5575765Z + echo 'Environment variables' 2025-11-03T16:27:55.5579568Z Environment variables 2025-11-03T16:27:55.5583637Z + env 2025-11-03T16:27:55.5590105Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-11-03T16:27:55.5592693Z CONTINUE_THROUGH_ERROR=True 2025-11-03T16:27:55.5592938Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-11-03T16:27:55.5593337Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-11-03T16:27:55.5593538Z HOSTNAME=06ea20fe7f0f 2025-11-03T16:27:55.5594185Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5594586Z GITHUB_ACTION=__run_3 2025-11-03T16:27:55.5594781Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-11-03T16:27:55.5594991Z GITHUB_RUN_NUMBER=163246 2025-11-03T16:27:55.5595204Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-11-03T16:27:55.5595426Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-11-03T16:27:55.5595640Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-11-03T16:27:55.5595846Z SCCACHE_IDLE_TIMEOUT=0 2025-11-03T16:27:55.5596145Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-11-03T16:27:55.5596346Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-11-03T16:27:55.5596540Z GITHUB_REF_TYPE=branch 2025-11-03T16:27:55.5596733Z BASE_SHA=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5596942Z XLA_CUDA= 2025-11-03T16:27:55.5597095Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-11-03T16:27:55.5597447Z HUGGING_FACE_HUB_TOKEN=*** 2025-11-03T16:27:55.5597678Z *** 2025-11-03T16:27:55.5597832Z GITHUB_REPOSITORY_ID=65600975 2025-11-03T16:27:55.5598012Z GITHUB_ACTIONS=true 2025-11-03T16:27:55.5598208Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-11-03T16:27:55.5598443Z SHA1=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5598671Z GITHUB_SHA=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5598981Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-11-03T16:27:55.5599263Z UCC_HOME=/usr 2025-11-03T16:27:55.5599410Z TORCH_SERIALIZATION_DEBUG=1 2025-11-03T16:27:55.5599582Z VERBOSE_TEST_LOGS=False 2025-11-03T16:27:55.5599751Z GITHUB_REF=refs/heads/main 2025-11-03T16:27:55.5599913Z SHARD_NUMBER=1 2025-11-03T16:27:55.5600059Z GITHUB_REF_PROTECTED=true 2025-11-03T16:27:55.5600226Z HOME=/var/lib/jenkins 2025-11-03T16:27:55.5600417Z GITHUB_API_URL=https://api.github.com 2025-11-03T16:27:55.5600626Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-11-03T16:27:55.5600802Z UCX_COMMIT= 2025-11-03T16:27:55.5600946Z USE_SYSTEM_NCCL=1 2025-11-03T16:27:55.5601099Z NUM_TEST_SHARDS=1 2025-11-03T16:27:55.5601265Z UCX_HOME=/usr 2025-11-03T16:27:55.5601585Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5602133Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:27:55.5602597Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5603050Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-11-03T16:27:55.5603341Z GITHUB_EVENT_NAME=push 2025-11-03T16:27:55.5603505Z DASHBOARD_TAG= 2025-11-03T16:27:55.5603662Z GITHUB_RUN_ID=19040285537 2025-11-03T16:27:55.5603835Z INSTALLED_OPENBLAS= 2025-11-03T16:27:55.5604189Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5604584Z GITHUB_ACTOR=pytorchmergebot 2025-11-03T16:27:55.5604763Z PR_NUMBER= 2025-11-03T16:27:55.5604910Z DESIRED_CUDA= 2025-11-03T16:27:55.5605053Z GITHUB_RUN_ATTEMPT=1 2025-11-03T16:27:55.5605214Z VALGRIND=ON 2025-11-03T16:27:55.5605363Z ANACONDA_PYTHON_VERSION=3.10 2025-11-03T16:27:55.5605575Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-11-03T16:27:55.5605778Z TERM=vt100 2025-11-03T16:27:55.5605919Z INSTALLED_VISION=yes 2025-11-03T16:27:55.5606073Z BRANCH=main 2025-11-03T16:27:55.5606222Z SCCACHE_REGION=us-east-1 2025-11-03T16:27:55.5606392Z OPENSSL_ROOT_DIR=/opt/openssl 2025-11-03T16:27:55.5606576Z BUILD_AOT_INDUCTOR_TEST=yes 2025-11-03T16:27:55.5606750Z CUDA_PATH=/usr/local/cuda 2025-11-03T16:27:55.5607060Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-11-03T16:27:55.5607390Z GITHUB_SERVER_URL=https://github.com 2025-11-03T16:27:55.5607581Z UCC_COMMIT= 2025-11-03T16:27:55.5607721Z REENABLED_ISSUES= 2025-11-03T16:27:55.5607915Z DOCS=yes 2025-11-03T16:27:55.5608048Z SHLVL=1 2025-11-03T16:27:55.5608185Z MAX_JOBS=30 2025-11-03T16:27:55.5608329Z GITHUB_ACTOR_ID=97764156 2025-11-03T16:27:55.5608546Z GITHUB_WORKFLOW_SHA=3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:27:55.5608768Z GITHUB_REF_NAME=main 2025-11-03T16:27:55.5609010Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-11-03T16:27:55.5609261Z GITHUB_JOB=test 2025-11-03T16:27:55.5609412Z NO_TEST_TIMEOUT=False 2025-11-03T16:27:55.5609564Z TD_DISTRIBUTED=False 2025-11-03T16:27:55.5609740Z GITHUB_REPOSITORY=pytorch/pytorch 2025-11-03T16:27:55.5609930Z GITHUB_RETENTION_DAYS=90 2025-11-03T16:27:55.5610095Z OPENSSL_DIR=/opt/openssl 2025-11-03T16:27:55.5610259Z GITHUB_ACTION_REPOSITORY= 2025-11-03T16:27:55.5610702Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-11-03T16:27:55.5611145Z GITHUB_BASE_REF= 2025-11-03T16:27:55.5611297Z INSTALLED_ACL= 2025-11-03T16:27:55.5611560Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:27:55.5611856Z CI=true 2025-11-03T16:27:55.5612002Z GITHUB_REPOSITORY_OWNER=pytorch 2025-11-03T16:27:55.5612214Z RUST_LOG=sccache::server=error 2025-11-03T16:27:55.5612378Z JOB_ID=54378387707 2025-11-03T16:27:55.5612526Z GITHUB_HEAD_REF= 2025-11-03T16:27:55.5612672Z GITHUB_ACTION_REF= 2025-11-03T16:27:55.5612858Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-11-03T16:27:55.5613067Z TEST_SHOWLOCALS=False 2025-11-03T16:27:55.5613436Z GITHUB_WORKFLOW=inductor 2025-11-03T16:27:55.5613626Z DEBIAN_FRONTEND=noninteractive 2025-11-03T16:27:55.5613986Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a026f0b7-be70-411b-be23-51115af743ea 2025-11-03T16:27:55.5614336Z NO_TD=False 2025-11-03T16:27:55.5614516Z SKIP_SCCACHE_INITIALIZATION=1 2025-11-03T16:27:55.5614718Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-11-03T16:27:55.5614925Z _=/usr/bin/env 2025-11-03T16:27:55.5615070Z + echo 'Testing pytorch' 2025-11-03T16:27:55.5615238Z Testing pytorch 2025-11-03T16:27:55.5615400Z + export LANG=C.UTF-8 2025-11-03T16:27:55.5615628Z + LANG=C.UTF-8 2025-11-03T16:27:55.5615766Z + PR_NUMBER= 2025-11-03T16:27:55.5615956Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-11-03T16:27:55.5616228Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-11-03T16:27:55.5616490Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-11-03T16:27:55.5616745Z + [[ linux-jammy-py3.10-gcc11-build == *slow-gradcheck* ]] 2025-11-03T16:27:55.5616999Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-11-03T16:27:55.5617229Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-11-03T16:27:55.5617457Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-11-03T16:27:55.5617689Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-11-03T16:27:55.5617919Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-11-03T16:27:55.5618143Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-11-03T16:27:55.5618371Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-11-03T16:27:55.5618585Z + pip_install ninja==1.10.2 2025-11-03T16:27:55.5618806Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-11-03T16:27:55.5619079Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-11-03T16:27:55.9358930Z Collecting ninja==1.10.2 2025-11-03T16:27:55.9523531Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-11-03T16:27:55.9645455Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-11-03T16:27:56.6584904Z Installing collected packages: ninja 2025-11-03T16:27:56.6587290Z Attempting uninstall: ninja 2025-11-03T16:27:56.6593684Z Found existing installation: ninja 1.11.1.4 2025-11-03T16:27:56.6607073Z Uninstalling ninja-1.11.1.4: 2025-11-03T16:27:56.6652536Z Successfully uninstalled ninja-1.11.1.4 2025-11-03T16:27:56.7110985Z Successfully installed ninja-1.10.2 2025-11-03T16:27:56.8099616Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-11-03T16:27:56.8100942Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-11-03T16:27:56.8101487Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-11-03T16:27:56.8101750Z + [[ linux-jammy-py3.10-gcc11-build == *asan* ]] 2025-11-03T16:27:56.8101997Z + [[ linux-jammy-py3.10-gcc11-build == *-debug* ]] 2025-11-03T16:27:56.8102239Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-11-03T16:27:56.8102568Z + echo 'We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass' 2025-11-03T16:27:56.8102981Z We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass 2025-11-03T16:27:56.8103252Z + cd test 2025-11-03T16:27:56.8103480Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-11-03T16:27:57.0919329Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:27:57.0920881Z import pynvml # type: ignore[import] 2025-11-03T16:27:57.8959669Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-11-03T16:27:57.8961324Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-11-03T16:27:57.8961804Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-11-03T16:27:57.8966504Z + DYNAMO_BENCHMARK_FLAGS=() 2025-11-03T16:27:57.8971809Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-11-03T16:27:57.8976600Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-11-03T16:27:57.8978681Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-11-03T16:27:57.8979450Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-11-03T16:27:57.8984198Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-11-03T16:27:57.8986562Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-11-03T16:27:57.8986870Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-11-03T16:27:57.8987134Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-11-03T16:27:57.8987364Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-11-03T16:27:57.8987654Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-11-03T16:27:57.8987929Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-11-03T16:27:57.8988138Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-11-03T16:27:57.9032882Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-11-03T16:27:57.9033385Z + [[ linux-jammy-py3.10-gcc11-build == *-bazel-* ]] 2025-11-03T16:27:57.9033705Z + cd test 2025-11-03T16:27:57.9034135Z + python -c 'import torch; print(torch.__config__.show())' 2025-11-03T16:27:58.1575953Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:27:58.1576836Z import pynvml # type: ignore[import] 2025-11-03T16:27:58.7314961Z PyTorch built with: 2025-11-03T16:27:58.7319117Z - GCC 11.4 2025-11-03T16:27:58.7323820Z - C++ Version: 201703 2025-11-03T16:27:58.7328514Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-11-03T16:27:58.7330501Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-11-03T16:27:58.7331252Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-11-03T16:27:58.7335615Z - LAPACK is enabled (usually provided by MKL) 2025-11-03T16:27:58.7337778Z - NNPACK is enabled 2025-11-03T16:27:58.7338129Z - CPU capability usage: AVX512 2025-11-03T16:27:58.7345292Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=3f6538febd727b782e6e13cfd026a309fb14351d, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-11-03T16:27:58.7347836Z 2025-11-03T16:27:58.9561039Z + cd test 2025-11-03T16:27:58.9561537Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-11-03T16:27:59.2420736Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:27:59.2421739Z import pynvml # type: ignore[import] 2025-11-03T16:27:59.8101661Z ATen/Parallel: 2025-11-03T16:27:59.8103389Z at::get_num_threads() : 16 2025-11-03T16:27:59.8103608Z at::get_num_interop_threads() : 16 2025-11-03T16:27:59.8103811Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-11-03T16:27:59.8104367Z omp_get_max_threads() : 16 2025-11-03T16:27:59.8104700Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-11-03T16:27:59.8105038Z mkl_get_max_threads() : 16 2025-11-03T16:27:59.8105284Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-11-03T16:27:59.8105554Z std::thread::hardware_concurrency() : 32 2025-11-03T16:27:59.8105754Z Environment variables: 2025-11-03T16:27:59.8105920Z OMP_NUM_THREADS : [not set] 2025-11-03T16:27:59.8106092Z MKL_NUM_THREADS : [not set] 2025-11-03T16:27:59.8106267Z ATen parallel backend: OpenMP 2025-11-03T16:27:59.8106379Z 2025-11-03T16:28:00.0250345Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-11-03T16:28:00.0250701Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-11-03T16:28:00.0251009Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-11-03T16:28:00.0251261Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-11-03T16:28:00.0251516Z + [[ dynamic_cpu_inductor_huggingface == *vllm* ]] 2025-11-03T16:28:00.0251763Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-11-03T16:28:00.0252038Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-11-03T16:28:00.0252330Z + [[ dynamic_cpu_inductor_huggingface == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-11-03T16:28:00.0252610Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-11-03T16:28:00.0252870Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-11-03T16:28:00.0253132Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-11-03T16:28:00.0253432Z + [[ dynamic_cpu_inductor_huggingface == *operator_microbenchmark* ]] 2025-11-03T16:28:00.0253736Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-11-03T16:28:00.0254395Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-11-03T16:28:00.0254692Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-11-03T16:28:00.0255005Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-11-03T16:28:00.0255437Z + [[ dynamic_cpu_inductor_huggingface == *aoti_cross_compile_for_windows* ]] 2025-11-03T16:28:00.0255808Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-11-03T16:28:00.0256038Z + install_torchvision 2025-11-03T16:28:00.0256203Z + local orig_preload 2025-11-03T16:28:00.0256367Z + local commit 2025-11-03T16:28:00.0256590Z ++ get_pinned_commit vision 2025-11-03T16:28:00.0256786Z ++ cat .github/ci_commit_pins/vision.txt 2025-11-03T16:28:00.0264291Z + commit=cfbc5c2f1c798991715a6b06bb3ce46478c4487c 2025-11-03T16:28:00.0264668Z + orig_preload= 2025-11-03T16:28:00.0264869Z + '[' -n '' ']' 2025-11-03T16:28:00.0265057Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-11-03T16:28:00.0265496Z + pip_build_and_install git+https://github.com/pytorch/vision.git@cfbc5c2f1c798991715a6b06bb3ce46478c4487c dist/vision 2025-11-03T16:28:00.0266021Z + local build_target=git+https://github.com/pytorch/vision.git@cfbc5c2f1c798991715a6b06bb3ce46478c4487c 2025-11-03T16:28:00.0266500Z + local wheel_dir=dist/vision 2025-11-03T16:28:00.0267538Z + local found_whl=0 2025-11-03T16:28:00.0268021Z + for file in "${wheel_dir}"/*.whl 2025-11-03T16:28:00.0268480Z + [[ -f dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl ]] 2025-11-03T16:28:00.0269354Z + found_whl=1 2025-11-03T16:28:00.0269605Z + break 2025-11-03T16:28:00.0269823Z + '[' 1 == 0 ']' 2025-11-03T16:28:00.0270036Z + for file in "${wheel_dir}"/*.whl 2025-11-03T16:28:00.0270472Z + pip_install_whl dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl 2025-11-03T16:28:00.0271006Z + args=('dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl') 2025-11-03T16:28:00.0271360Z + local args 2025-11-03T16:28:00.0271728Z + [[ dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl == *\ * ]] 2025-11-03T16:28:00.0272094Z + for path in "${args[@]}" 2025-11-03T16:28:00.0272450Z + echo 'Installing dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl' 2025-11-03T16:28:00.0273308Z Installing dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl 2025-11-03T16:28:00.0273868Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl 2025-11-03T16:28:00.2988431Z Processing ./dist/vision/torchvision-0.25.0a0+cfbc5c2-cp310-cp310-linux_x86_64.whl 2025-11-03T16:28:00.3062070Z Installing collected packages: torchvision 2025-11-03T16:28:00.8928797Z Successfully installed torchvision-0.25.0a0+cfbc5c2 2025-11-03T16:28:00.9374816Z + '[' -n '' ']' 2025-11-03T16:28:00.9375166Z + id=0 2025-11-03T16:28:00.9381213Z + test_dynamo_benchmark huggingface 0 2025-11-03T16:28:00.9386729Z ++ pwd 2025-11-03T16:28:00.9388920Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-11-03T16:28:00.9389416Z + local suite=huggingface 2025-11-03T16:28:00.9389697Z + shift 2025-11-03T16:28:00.9389931Z + local shard_id=0 2025-11-03T16:28:00.9390083Z + shift 2025-11-03T16:28:00.9390420Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-11-03T16:28:00.9390724Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-11-03T16:28:00.9391533Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-11-03T16:28:00.9391792Z + local dt=float32 2025-11-03T16:28:00.9392000Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-11-03T16:28:00.9392253Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-11-03T16:28:00.9392570Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-11-03T16:28:00.9392843Z ++ pwd 2025-11-03T16:28:00.9393048Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-11-03T16:28:00.9393351Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-11-03T16:28:00.9404305Z + local name=inference 2025-11-03T16:28:00.9409073Z + shift 2025-11-03T16:28:00.9414385Z + local suite=huggingface 2025-11-03T16:28:00.9418547Z + shift 2025-11-03T16:28:00.9420588Z + local shard_id=0 2025-11-03T16:28:00.9420876Z + shift 2025-11-03T16:28:00.9426332Z + partition_flags=() 2025-11-03T16:28:00.9428504Z + local partition_flags 2025-11-03T16:28:00.9428736Z + [[ -n 1 ]] 2025-11-03T16:28:00.9428889Z + [[ -n 0 ]] 2025-11-03T16:28:00.9429157Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-11-03T16:28:00.9429502Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-11-03T16:28:00.9429749Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-11-03T16:28:00.9429988Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-11-03T16:28:00.9430232Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-11-03T16:28:00.9431049Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-11-03T16:28:01.6402638Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:28:01.6403633Z import pynvml # type: ignore[import] 2025-11-03T16:28:04.7729600Z 2025-11-03T16:28:04.7734163Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-11-03T16:29:30.4933411Z 2025-11-03T16:29:30.4933513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4934022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4934504Z layer_outputs = layer_module( 2025-11-03T16:29:30.4934847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4935216Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4935641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4936063Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4936957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4937411Z self_outputs = self.self( 2025-11-03T16:29:30.4937815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.4938258Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.4938794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.4939384Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.4939628Z 2025-11-03T16:29:30.4939732Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4940252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4940735Z layer_outputs = layer_module( 2025-11-03T16:29:30.4941086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4941459Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4941876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4942300Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4942723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4943136Z self_outputs = self.self( 2025-11-03T16:29:30.4943540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.4943978Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.4944479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.4945116Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.4945364Z 2025-11-03T16:29:30.4945478Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4946005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4946499Z layer_outputs = layer_module( 2025-11-03T16:29:30.4946859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4947236Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4947677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4948096Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4948515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4948935Z self_outputs = self.self( 2025-11-03T16:29:30.4949350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.4949813Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.4950321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.4950924Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.4951176Z 2025-11-03T16:29:30.4951261Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.4951482Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.4951701Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.4951902Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.4952144Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4952689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4953222Z layer_outputs = layer_module( 2025-11-03T16:29:30.4953589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4954042Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4954505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4954950Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4955392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4955855Z self_outputs = self.self( 2025-11-03T16:29:30.4956250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.4956691Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.4957188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.4957715Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.4958225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.4958695Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.4958852Z 2025-11-03T16:29:30.4958930Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.4959161Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4959710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4960179Z layer_outputs = layer_module( 2025-11-03T16:29:30.4960530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4960889Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4961301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4961711Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4962115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4962516Z self_outputs = self.self( 2025-11-03T16:29:30.4962912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.4963318Z attn_scores += diagonal_mask 2025-11-03T16:29:30.4963441Z 2025-11-03T16:29:30.4963550Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4964046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4964526Z layer_outputs = layer_module( 2025-11-03T16:29:30.4964868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4965225Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4965634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4966038Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4966451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4966850Z self_outputs = self.self( 2025-11-03T16:29:30.4967243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.4967682Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.4967820Z 2025-11-03T16:29:30.4967920Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4968418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4968886Z layer_outputs = layer_module( 2025-11-03T16:29:30.4969224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4969575Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4969989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4970392Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4970799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4971195Z self_outputs = self.self( 2025-11-03T16:29:30.4971573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.4972017Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.4972529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.4973100Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.4973552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.4973898Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.4974059Z 2025-11-03T16:29:30.4974162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4974665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4975137Z layer_outputs = layer_module( 2025-11-03T16:29:30.4975475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4975824Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4976237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4976645Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4977058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4977457Z self_outputs = self.self( 2025-11-03T16:29:30.4977843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.4978290Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.4978807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.4979336Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.4979838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.4980291Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.4980632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.4980976Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.4981166Z 2025-11-03T16:29:30.4981277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4981783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4982255Z layer_outputs = layer_module( 2025-11-03T16:29:30.4982607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4982982Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4983415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4983829Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4984265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4984677Z self_outputs = self.self( 2025-11-03T16:29:30.4985076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.4985526Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.4986033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.4986588Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.4986800Z 2025-11-03T16:29:30.4986901Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4987439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4987910Z layer_outputs = layer_module( 2025-11-03T16:29:30.4988247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4988604Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4989012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4989418Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4989827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4990220Z self_outputs = self.self( 2025-11-03T16:29:30.4990608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.4991053Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.4991563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.4992108Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.4992310Z 2025-11-03T16:29:30.4992413Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4992926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4993408Z layer_outputs = layer_module( 2025-11-03T16:29:30.4993756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4994221Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4994645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4995068Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.4995530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.4995949Z self_outputs = self.self( 2025-11-03T16:29:30.4996340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.4996849Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.4997092Z 2025-11-03T16:29:30.4997196Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.4997688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.4998157Z layer_outputs = layer_module( 2025-11-03T16:29:30.4998495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.4998848Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.4999253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.4999656Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5000059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5000496Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5000928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5001348Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5001542Z 2025-11-03T16:29:30.5001642Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5002130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5002594Z layer_outputs = layer_module( 2025-11-03T16:29:30.5002919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5003268Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5003673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5004090Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5004463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5004842Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5005251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5005688Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5006106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5006493Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5006631Z 2025-11-03T16:29:30.5006727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5007203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5007655Z layer_outputs = layer_module( 2025-11-03T16:29:30.5007983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5008318Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5008713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5009142Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5009526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5009905Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5010297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5010737Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5011150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5011573Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5011931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5012248Z return self.act(input) 2025-11-03T16:29:30.5012360Z 2025-11-03T16:29:30.5012456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5012929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5013579Z layer_outputs = layer_module( 2025-11-03T16:29:30.5013919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5014265Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5014672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5015156Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5015537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5015901Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5016282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5016712Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5017139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5017532Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5017661Z 2025-11-03T16:29:30.5017756Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5018235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5018683Z layer_outputs = layer_module( 2025-11-03T16:29:30.5019006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5019348Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5019726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5020109Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5020497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5020873Z self_outputs = self.self( 2025-11-03T16:29:30.5021240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5021626Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5021761Z 2025-11-03T16:29:30.5021855Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5022377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5022832Z layer_outputs = layer_module( 2025-11-03T16:29:30.5023159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5023493Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5023888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5024278Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5024669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5025054Z self_outputs = self.self( 2025-11-03T16:29:30.5025427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5025849Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5026321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5026866Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5027091Z 2025-11-03T16:29:30.5027201Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5027671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5028168Z layer_outputs = layer_module( 2025-11-03T16:29:30.5028507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5028858Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5029257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5029647Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5030043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5030436Z self_outputs = self.self( 2025-11-03T16:29:30.5030817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5031209Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5031344Z 2025-11-03T16:29:30.5031443Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5031926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5032385Z layer_outputs = layer_module( 2025-11-03T16:29:30.5032718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5033057Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5033463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5033864Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5034332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5034737Z self_outputs = self.self( 2025-11-03T16:29:30.5035126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5035571Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5036117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5036682Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5036918Z 2025-11-03T16:29:30.5037029Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5037519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5037985Z layer_outputs = layer_module( 2025-11-03T16:29:30.5038329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5038711Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5039120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5039526Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5039937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5040340Z self_outputs = self.self( 2025-11-03T16:29:30.5040730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5041158Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5041634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5042231Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5042469Z 2025-11-03T16:29:30.5042574Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5043064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5043524Z layer_outputs = layer_module( 2025-11-03T16:29:30.5043851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5044200Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5044601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5045008Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5045407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5045791Z self_outputs = self.self( 2025-11-03T16:29:30.5046174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5046593Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5047067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5047618Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5047843Z 2025-11-03T16:29:30.5047923Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5048131Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5048331Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5048529Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5048749Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5049240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5049748Z layer_outputs = layer_module( 2025-11-03T16:29:30.5050079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5050432Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5050829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5051231Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5051640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5052037Z self_outputs = self.self( 2025-11-03T16:29:30.5052416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5052851Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5053334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5053846Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5054349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5054813Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5054954Z 2025-11-03T16:29:30.5055027Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5055247Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5055755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5056213Z layer_outputs = layer_module( 2025-11-03T16:29:30.5056533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5056874Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5057268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5057658Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5058046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5058482Z self_outputs = self.self( 2025-11-03T16:29:30.5058870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5059271Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5059383Z 2025-11-03T16:29:30.5059491Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5059967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5060413Z layer_outputs = layer_module( 2025-11-03T16:29:30.5060750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5061098Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5061499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5061897Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5062292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5062684Z self_outputs = self.self( 2025-11-03T16:29:30.5063099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5063501Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5063630Z 2025-11-03T16:29:30.5063735Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5064216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5064678Z layer_outputs = layer_module( 2025-11-03T16:29:30.5065012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5065368Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5065764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5066164Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5066559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5066958Z self_outputs = self.self( 2025-11-03T16:29:30.5067336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5067739Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5067880Z 2025-11-03T16:29:30.5067978Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5068469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5068961Z layer_outputs = layer_module( 2025-11-03T16:29:30.5069304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5069648Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5070047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5070441Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5070833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5071222Z self_outputs = self.self( 2025-11-03T16:29:30.5071590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5072024Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5072524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5073083Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5073487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5073821Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5074041Z 2025-11-03T16:29:30.5074149Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5074661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5075140Z layer_outputs = layer_module( 2025-11-03T16:29:30.5075486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5075846Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5076260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5076715Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5077130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5077531Z self_outputs = self.self( 2025-11-03T16:29:30.5077927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5078384Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5078936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5079479Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5079982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5080454Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5080801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5081161Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5081314Z 2025-11-03T16:29:30.5081425Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5081924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5082401Z layer_outputs = layer_module( 2025-11-03T16:29:30.5082784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5083151Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5083587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5083978Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5084386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5084781Z self_outputs = self.self( 2025-11-03T16:29:30.5085162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5085594Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5086086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5086623Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5086831Z 2025-11-03T16:29:30.5086930Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5087419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5087873Z layer_outputs = layer_module( 2025-11-03T16:29:30.5088193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5088549Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5088933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5089319Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5089705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5090077Z self_outputs = self.self( 2025-11-03T16:29:30.5090483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5090903Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5091388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5091905Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5092091Z 2025-11-03T16:29:30.5092188Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5092667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5093116Z layer_outputs = layer_module( 2025-11-03T16:29:30.5093440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5093779Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5094159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5094545Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5094936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5095320Z self_outputs = self.self( 2025-11-03T16:29:30.5095694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5096206Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5096439Z 2025-11-03T16:29:30.5096535Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5097009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5097461Z layer_outputs = layer_module( 2025-11-03T16:29:30.5097780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5098111Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5098497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5098881Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5099262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5099681Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5100090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5100488Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5100630Z 2025-11-03T16:29:30.5100726Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5101207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5101675Z layer_outputs = layer_module( 2025-11-03T16:29:30.5101994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5102333Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5102723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5103117Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5103532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5103902Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5104286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5104711Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5105132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5105517Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5105653Z 2025-11-03T16:29:30.5105748Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5106227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5106684Z layer_outputs = layer_module( 2025-11-03T16:29:30.5107012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5107350Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5107743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5108142Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5108526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5108904Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5109322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5109760Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5110198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5110638Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5111014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5111347Z return self.act(input) 2025-11-03T16:29:30.5111464Z 2025-11-03T16:29:30.5111563Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5112066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5112534Z layer_outputs = layer_module( 2025-11-03T16:29:30.5112868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5113343Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5113773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5114273Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5114704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5115113Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5115529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5116022Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5116498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5116927Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5117065Z 2025-11-03T16:29:30.5117228Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5117740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5118218Z layer_outputs = layer_module( 2025-11-03T16:29:30.5118560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5118916Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5119341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5119767Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5120205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5120624Z self_outputs = self.self( 2025-11-03T16:29:30.5121006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5121393Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5121527Z 2025-11-03T16:29:30.5121622Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5122102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5122556Z layer_outputs = layer_module( 2025-11-03T16:29:30.5122881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5123210Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5123658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5124047Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5124433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5124807Z self_outputs = self.self( 2025-11-03T16:29:30.5125182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5125600Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5126064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5126603Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5126828Z 2025-11-03T16:29:30.5126922Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5127397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5127845Z layer_outputs = layer_module( 2025-11-03T16:29:30.5128165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5128503Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5128883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5129264Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5129649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5130029Z self_outputs = self.self( 2025-11-03T16:29:30.5130398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5130818Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5130950Z 2025-11-03T16:29:30.5131044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5131521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5131965Z layer_outputs = layer_module( 2025-11-03T16:29:30.5132286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5132615Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5133003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5133394Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5133782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5134163Z self_outputs = self.self( 2025-11-03T16:29:30.5134525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5134939Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5135407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5135944Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5136167Z 2025-11-03T16:29:30.5136269Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5136765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5137216Z layer_outputs = layer_module( 2025-11-03T16:29:30.5137542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5137880Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5138271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5138651Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5139037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5139416Z self_outputs = self.self( 2025-11-03T16:29:30.5139787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5140194Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5140664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5141209Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5141443Z 2025-11-03T16:29:30.5141539Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5142027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5142475Z layer_outputs = layer_module( 2025-11-03T16:29:30.5142794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5143135Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5143557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5143977Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5144362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5144739Z self_outputs = self.self( 2025-11-03T16:29:30.5145106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5145509Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5145967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5146499Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5146731Z 2025-11-03T16:29:30.5146806Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5147006Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5147206Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5147398Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5147606Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5148085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5148530Z layer_outputs = layer_module( 2025-11-03T16:29:30.5148852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5149179Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5149612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5150000Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5150386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5150767Z self_outputs = self.self( 2025-11-03T16:29:30.5151132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5151550Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5152024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5152535Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5153036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5153485Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5153636Z 2025-11-03T16:29:30.5153713Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5153940Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5154513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5154992Z layer_outputs = layer_module( 2025-11-03T16:29:30.5155330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5155695Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5156121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5156527Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5156929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5157358Z self_outputs = self.self( 2025-11-03T16:29:30.5157738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5158139Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5158255Z 2025-11-03T16:29:30.5158358Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5158842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5159294Z layer_outputs = layer_module( 2025-11-03T16:29:30.5159623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5159971Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5160372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5160770Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5161165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5161554Z self_outputs = self.self( 2025-11-03T16:29:30.5161930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5162327Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5162455Z 2025-11-03T16:29:30.5162551Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5163064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5163529Z layer_outputs = layer_module( 2025-11-03T16:29:30.5163858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5164207Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5164598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5164993Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5165387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5165780Z self_outputs = self.self( 2025-11-03T16:29:30.5166158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5166555Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5166699Z 2025-11-03T16:29:30.5166797Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5167281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5167738Z layer_outputs = layer_module( 2025-11-03T16:29:30.5168069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5168411Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5168815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5169216Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5169613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5170001Z self_outputs = self.self( 2025-11-03T16:29:30.5170380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5170862Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5171375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5171942Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5172361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5172705Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5172871Z 2025-11-03T16:29:30.5172973Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5173473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5173947Z layer_outputs = layer_module( 2025-11-03T16:29:30.5174286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5174642Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5175057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5175465Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5175872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5176267Z self_outputs = self.self( 2025-11-03T16:29:30.5176654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5177135Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5177653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5178188Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5178685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5179144Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5179482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5179828Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5179975Z 2025-11-03T16:29:30.5180082Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5180579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5181056Z layer_outputs = layer_module( 2025-11-03T16:29:30.5181396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5181752Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5182162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5182564Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5182976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5183388Z self_outputs = self.self( 2025-11-03T16:29:30.5183780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5184314Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5184821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5185405Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5185614Z 2025-11-03T16:29:30.5185719Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5186225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5186702Z layer_outputs = layer_module( 2025-11-03T16:29:30.5187040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5187402Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5187822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5188240Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5188655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5189058Z self_outputs = self.self( 2025-11-03T16:29:30.5189454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5189908Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5190425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5191017Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5191213Z 2025-11-03T16:29:30.5191310Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5191798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5192259Z layer_outputs = layer_module( 2025-11-03T16:29:30.5192588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5192937Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5193339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5193743Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5194200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5194603Z self_outputs = self.self( 2025-11-03T16:29:30.5194981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5195488Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5195726Z 2025-11-03T16:29:30.5195825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5196316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5196774Z layer_outputs = layer_module( 2025-11-03T16:29:30.5197099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5197450Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5197853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5198249Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5198687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5199118Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5199555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5199963Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5200098Z 2025-11-03T16:29:30.5200206Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5200700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5201168Z layer_outputs = layer_module( 2025-11-03T16:29:30.5201507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5201859Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5202264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5202674Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5203061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5203447Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5203851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5204293Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5204774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5205176Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5205317Z 2025-11-03T16:29:30.5205417Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5205905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5206365Z layer_outputs = layer_module( 2025-11-03T16:29:30.5206695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5207040Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5207443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5207838Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5208217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5208585Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5208972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5209396Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5209814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5210234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5210588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5210909Z return self.act(input) 2025-11-03T16:29:30.5211019Z 2025-11-03T16:29:30.5211117Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5211594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5212568Z layer_outputs = layer_module( 2025-11-03T16:29:30.5212882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5213367Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5213768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5214166Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5214546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5214911Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5215307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5215742Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5216178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5216573Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5216700Z 2025-11-03T16:29:30.5216796Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5217275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5217720Z layer_outputs = layer_module( 2025-11-03T16:29:30.5218043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5218445Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5218832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5219229Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5219621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5220010Z self_outputs = self.self( 2025-11-03T16:29:30.5220379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5220777Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5220912Z 2025-11-03T16:29:30.5221006Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5221496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5221954Z layer_outputs = layer_module( 2025-11-03T16:29:30.5222275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5222615Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5223001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5223390Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5223778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5224156Z self_outputs = self.self( 2025-11-03T16:29:30.5224528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5224945Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5225419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5226003Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5226230Z 2025-11-03T16:29:30.5226326Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5226802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5227248Z layer_outputs = layer_module( 2025-11-03T16:29:30.5227570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5227907Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5228293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5228677Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5229063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5229449Z self_outputs = self.self( 2025-11-03T16:29:30.5229822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5230203Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5230334Z 2025-11-03T16:29:30.5230429Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5230901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5231349Z layer_outputs = layer_module( 2025-11-03T16:29:30.5231707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5232041Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5232433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5232818Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5233202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5233575Z self_outputs = self.self( 2025-11-03T16:29:30.5234000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5234437Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5234925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5235477Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5235708Z 2025-11-03T16:29:30.5235812Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5236288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5236750Z layer_outputs = layer_module( 2025-11-03T16:29:30.5237079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5237430Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5237820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5238217Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5238614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5239045Z self_outputs = self.self( 2025-11-03T16:29:30.5239424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5239845Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5240319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5240865Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5241089Z 2025-11-03T16:29:30.5241195Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5241683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5242134Z layer_outputs = layer_module( 2025-11-03T16:29:30.5242463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5242813Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5243211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5243607Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5243995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5244387Z self_outputs = self.self( 2025-11-03T16:29:30.5244769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5245230Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5245704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5246247Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5246481Z 2025-11-03T16:29:30.5246561Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5246768Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5246977Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5247176Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5247391Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5247877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5248332Z layer_outputs = layer_module( 2025-11-03T16:29:30.5248673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5249006Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5249398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5249784Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5250171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5250551Z self_outputs = self.self( 2025-11-03T16:29:30.5250910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5251331Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5251805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5252304Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5252842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5253280Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5253428Z 2025-11-03T16:29:30.5253501Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5253722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5254198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5254644Z layer_outputs = layer_module( 2025-11-03T16:29:30.5254967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5255308Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5255699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5256091Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5256418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5256544Z self_outputs = self.self( 2025-11-03T16:29:30.5257001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5257109Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5257113Z 2025-11-03T16:29:30.5257272Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5257660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5257757Z layer_outputs = layer_module( 2025-11-03T16:29:30.5258003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5258128Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5258451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5258568Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5258875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5258962Z self_outputs = self.self( 2025-11-03T16:29:30.5259226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5259385Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5259389Z 2025-11-03T16:29:30.5259505Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5259877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5259964Z layer_outputs = layer_module( 2025-11-03T16:29:30.5260208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5260321Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5260618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5260736Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5261021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5261137Z self_outputs = self.self( 2025-11-03T16:29:30.5261403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5261619Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5261623Z 2025-11-03T16:29:30.5261741Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5262085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5262200Z layer_outputs = layer_module( 2025-11-03T16:29:30.5262426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5262565Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5262878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5262995Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5263275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5263363Z self_outputs = self.self( 2025-11-03T16:29:30.5263665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5263812Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5264195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5264377Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5264612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5264764Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5264768Z 2025-11-03T16:29:30.5264875Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5265276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5265366Z layer_outputs = layer_module( 2025-11-03T16:29:30.5265631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5265729Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5266017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5266141Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5266438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5266554Z self_outputs = self.self( 2025-11-03T16:29:30.5266850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5267011Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5267358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5267542Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5267902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5268016Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5268254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5268370Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5268403Z 2025-11-03T16:29:30.5268555Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5268920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5269017Z layer_outputs = layer_module( 2025-11-03T16:29:30.5269277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5269378Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5287099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5287280Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5287640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5287727Z self_outputs = self.self( 2025-11-03T16:29:30.5288029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5288165Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5288527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5288681Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5288698Z 2025-11-03T16:29:30.5288817Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5289284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5289377Z layer_outputs = layer_module( 2025-11-03T16:29:30.5289602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5289697Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5289978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5290060Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5290346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5290417Z self_outputs = self.self( 2025-11-03T16:29:30.5290701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5290823Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5291180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5291333Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5291337Z 2025-11-03T16:29:30.5291445Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5291801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5291876Z layer_outputs = layer_module( 2025-11-03T16:29:30.5292110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5292192Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5292479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5292558Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5292884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5292965Z self_outputs = self.self( 2025-11-03T16:29:30.5293243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5293449Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5293453Z 2025-11-03T16:29:30.5293553Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5293893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5293963Z layer_outputs = layer_module( 2025-11-03T16:29:30.5294176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5294262Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5294526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5294606Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5294873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5294980Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5295255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5295336Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5295340Z 2025-11-03T16:29:30.5295475Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5295803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5295879Z layer_outputs = layer_module( 2025-11-03T16:29:30.5296085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5296159Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5296430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5296510Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5296762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5296835Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5297103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5297220Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5297482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5297566Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5297569Z 2025-11-03T16:29:30.5297665Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5297998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5298064Z layer_outputs = layer_module( 2025-11-03T16:29:30.5298269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5298352Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5298613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5298745Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5298991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5299064Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5299338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5299442Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5299715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5299825Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5300038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5300108Z return self.act(input) 2025-11-03T16:29:30.5300112Z 2025-11-03T16:29:30.5300206Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5300542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5300610Z layer_outputs = layer_module( 2025-11-03T16:29:30.5300826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5300900Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5301170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5301277Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5301519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5301600Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5301862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5301984Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5302244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5302320Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5302331Z 2025-11-03T16:29:30.5302425Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5302752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5302827Z layer_outputs = layer_module( 2025-11-03T16:29:30.5303034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5303115Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5303375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5303445Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5303711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5303776Z self_outputs = self.self( 2025-11-03T16:29:30.5304043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5304122Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5304125Z 2025-11-03T16:29:30.5304226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5304582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5304651Z layer_outputs = layer_module( 2025-11-03T16:29:30.5304864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5304935Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5305203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5305275Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5305532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5305609Z self_outputs = self.self( 2025-11-03T16:29:30.5305869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5305976Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5306294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5306475Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5306479Z 2025-11-03T16:29:30.5306574Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5306898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5306972Z layer_outputs = layer_module( 2025-11-03T16:29:30.5307206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5307291Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5307554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5307629Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5307889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5307954Z self_outputs = self.self( 2025-11-03T16:29:30.5308221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5308295Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5308298Z 2025-11-03T16:29:30.5308399Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5308727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5308796Z layer_outputs = layer_module( 2025-11-03T16:29:30.5309008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5309080Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5309348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5309416Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5309683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5309746Z self_outputs = self.self( 2025-11-03T16:29:30.5310009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5310113Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5310456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5310630Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5310633Z 2025-11-03T16:29:30.5310728Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5311056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5311121Z layer_outputs = layer_module( 2025-11-03T16:29:30.5311322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5311405Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5311663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5311741Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5311998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5312061Z self_outputs = self.self( 2025-11-03T16:29:30.5312325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5312418Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5312737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5312970Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5312974Z 2025-11-03T16:29:30.5313077Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5313564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5313638Z layer_outputs = layer_module( 2025-11-03T16:29:30.5313864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5313941Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5314302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5314378Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5314668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5314740Z self_outputs = self.self( 2025-11-03T16:29:30.5315024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5315131Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5315454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5315632Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5315635Z 2025-11-03T16:29:30.5315716Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5315795Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5315880Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5315955Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5316065Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5316421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5316560Z layer_outputs = layer_module( 2025-11-03T16:29:30.5316780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5316861Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5317152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5317229Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5317521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5317592Z self_outputs = self.self( 2025-11-03T16:29:30.5317881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5318003Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5318353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5318510Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5318841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5318953Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5318957Z 2025-11-03T16:29:30.5319040Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5319147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5319569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5319647Z layer_outputs = layer_module( 2025-11-03T16:29:30.5319877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5319956Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5320237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5320320Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5320603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5320677Z self_outputs = self.self( 2025-11-03T16:29:30.5320961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5321044Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5321048Z 2025-11-03T16:29:30.5321148Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5321506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5321586Z layer_outputs = layer_module( 2025-11-03T16:29:30.5321811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5321896Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5322179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5322254Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5322550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5322618Z self_outputs = self.self( 2025-11-03T16:29:30.5322908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5323021Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5323025Z 2025-11-03T16:29:30.5323137Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5323490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5323560Z layer_outputs = layer_module( 2025-11-03T16:29:30.5323795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5323877Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5324169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5324237Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5324500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5324571Z self_outputs = self.self( 2025-11-03T16:29:30.5324831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5324919Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5324922Z 2025-11-03T16:29:30.5325018Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5325350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5325416Z layer_outputs = layer_module( 2025-11-03T16:29:30.5325651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5325735Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5325999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5326076Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5326336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5326407Z self_outputs = self.self( 2025-11-03T16:29:30.5326667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5326778Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5327113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5327277Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5327468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5327564Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5327567Z 2025-11-03T16:29:30.5327666Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5327989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5328054Z layer_outputs = layer_module( 2025-11-03T16:29:30.5328266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5328340Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5328603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5328704Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5328962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5329034Z self_outputs = self.self( 2025-11-03T16:29:30.5329291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5329406Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5329731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5329871Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5330168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5330257Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5330446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5330537Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5330540Z 2025-11-03T16:29:30.5330644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5330968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5331041Z layer_outputs = layer_module( 2025-11-03T16:29:30.5331245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5331350Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5331618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5331690Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5331954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5332020Z self_outputs = self.self( 2025-11-03T16:29:30.5332283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5332397Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5332720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5332869Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5332873Z 2025-11-03T16:29:30.5332966Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5333302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5333368Z layer_outputs = layer_module( 2025-11-03T16:29:30.5333573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5333656Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5333912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5333988Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5334247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5334313Z self_outputs = self.self( 2025-11-03T16:29:30.5334575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5334715Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5335048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5335189Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5335192Z 2025-11-03T16:29:30.5335293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5335623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5335693Z layer_outputs = layer_module( 2025-11-03T16:29:30.5335905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5335983Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5336254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5336323Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5336586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5336650Z self_outputs = self.self( 2025-11-03T16:29:30.5336910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5337094Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5337150Z 2025-11-03T16:29:30.5337246Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5337576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5337645Z layer_outputs = layer_module( 2025-11-03T16:29:30.5337855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5337928Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5338186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5338262Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5338523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5338638Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5338897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5338978Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5338988Z 2025-11-03T16:29:30.5339081Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5339405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5339480Z layer_outputs = layer_module( 2025-11-03T16:29:30.5339685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5339766Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5340027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5340107Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5340360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5340463Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5340730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5340833Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5341089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5341173Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5341176Z 2025-11-03T16:29:30.5341271Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5341606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5341672Z layer_outputs = layer_module( 2025-11-03T16:29:30.5341884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5341955Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5342212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5342299Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5342545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5342618Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5342923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5343027Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5343292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5343401Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5343607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5343673Z return self.act(input) 2025-11-03T16:29:30.5343677Z 2025-11-03T16:29:30.5343773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5344102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5344170Z layer_outputs = layer_module( 2025-11-03T16:29:30.5344386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5344457Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5344716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5344803Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5345045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5345122Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5345380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5345501Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5345757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5345834Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5345837Z 2025-11-03T16:29:30.5345936Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5346291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5346363Z layer_outputs = layer_module( 2025-11-03T16:29:30.5346568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5346646Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5346903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5346974Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5347239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5347305Z self_outputs = self.self( 2025-11-03T16:29:30.5347570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5347649Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5347653Z 2025-11-03T16:29:30.5347746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5348074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5348140Z layer_outputs = layer_module( 2025-11-03T16:29:30.5348350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5348422Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5348715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5348787Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5349049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5349123Z self_outputs = self.self( 2025-11-03T16:29:30.5349379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5349478Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5349792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5349968Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5349971Z 2025-11-03T16:29:30.5350068Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5350391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5350468Z layer_outputs = layer_module( 2025-11-03T16:29:30.5350671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5350750Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5351006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5351075Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5351338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5351402Z self_outputs = self.self( 2025-11-03T16:29:30.5351668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5351770Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5351774Z 2025-11-03T16:29:30.5351872Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5352196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5352260Z layer_outputs = layer_module( 2025-11-03T16:29:30.5352471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5352544Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5352811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5352882Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5353149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5353216Z self_outputs = self.self( 2025-11-03T16:29:30.5353475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5353576Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5353892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5354137Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5354143Z 2025-11-03T16:29:30.5354239Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5354605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5354681Z layer_outputs = layer_module( 2025-11-03T16:29:30.5354894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5354977Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5355244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5355322Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5355589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5355654Z self_outputs = self.self( 2025-11-03T16:29:30.5355931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5356029Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5356358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5356541Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5356544Z 2025-11-03T16:29:30.5356645Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5356970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5357038Z layer_outputs = layer_module( 2025-11-03T16:29:30.5357255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5357330Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5357610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5357711Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5357978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5358052Z self_outputs = self.self( 2025-11-03T16:29:30.5358317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5358421Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5358743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5358921Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5358926Z 2025-11-03T16:29:30.5359003Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5359078Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5359163Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5359236Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5359340Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5359674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5359741Z layer_outputs = layer_module( 2025-11-03T16:29:30.5359965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5360039Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5360314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5360418Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5360692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5360761Z self_outputs = self.self( 2025-11-03T16:29:30.5361025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5361139Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5361459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5361603Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5361907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5362009Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5362012Z 2025-11-03T16:29:30.5362089Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5362187Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5362528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5362596Z layer_outputs = layer_module( 2025-11-03T16:29:30.5362814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5362888Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5363154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5363235Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5363504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5363576Z self_outputs = self.self( 2025-11-03T16:29:30.5363883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5363961Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5363964Z 2025-11-03T16:29:30.5364061Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5364393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5364467Z layer_outputs = layer_module( 2025-11-03T16:29:30.5364677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5364757Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5365026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5365100Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5365373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5365439Z self_outputs = self.self( 2025-11-03T16:29:30.5365711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5365785Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5365789Z 2025-11-03T16:29:30.5365892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5366227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5366323Z layer_outputs = layer_module( 2025-11-03T16:29:30.5366540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5366618Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5366890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5366960Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5367226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5367300Z self_outputs = self.self( 2025-11-03T16:29:30.5367566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5367653Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5367657Z 2025-11-03T16:29:30.5367755Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5368094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5368162Z layer_outputs = layer_module( 2025-11-03T16:29:30.5368371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5368454Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5368720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5368797Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5369069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5369131Z self_outputs = self.self( 2025-11-03T16:29:30.5369397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5369536Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5369868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5370028Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5370221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5370314Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5370317Z 2025-11-03T16:29:30.5370410Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5370743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5370808Z layer_outputs = layer_module( 2025-11-03T16:29:30.5371020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5371094Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5371357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5371424Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5371680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5371751Z self_outputs = self.self( 2025-11-03T16:29:30.5372008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5372148Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5372477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5372614Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5372910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5372995Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5373184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5373275Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5373278Z 2025-11-03T16:29:30.5373380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5373709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5373774Z layer_outputs = layer_module( 2025-11-03T16:29:30.5373988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5374060Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5374325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5374394Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5374660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5374725Z self_outputs = self.self( 2025-11-03T16:29:30.5374983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5375099Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5375424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5375607Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5375610Z 2025-11-03T16:29:30.5375705Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5376039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5376105Z layer_outputs = layer_module( 2025-11-03T16:29:30.5376309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5376391Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5376653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5376731Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5376993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5377056Z self_outputs = self.self( 2025-11-03T16:29:30.5377321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5377428Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5377759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5377898Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5377935Z 2025-11-03T16:29:30.5378039Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5378362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5378430Z layer_outputs = layer_module( 2025-11-03T16:29:30.5378642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5378715Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5378978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5379048Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5379309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5379374Z self_outputs = self.self( 2025-11-03T16:29:30.5379628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5379811Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5379814Z 2025-11-03T16:29:30.5379907Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5380233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5380298Z layer_outputs = layer_module( 2025-11-03T16:29:30.5380505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5380577Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5380836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5380913Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5381167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5381314Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5381573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5381652Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5381655Z 2025-11-03T16:29:30.5381756Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5382081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5382153Z layer_outputs = layer_module( 2025-11-03T16:29:30.5382365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5382445Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5382714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5382794Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5383053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5383125Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5383399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5383503Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5383800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5383888Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5383892Z 2025-11-03T16:29:30.5383992Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5384342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5384407Z layer_outputs = layer_module( 2025-11-03T16:29:30.5384618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5384689Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5384952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5385036Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5385285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5385364Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5385629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5385736Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5385996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5386100Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5386309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5386377Z return self.act(input) 2025-11-03T16:29:30.5386380Z 2025-11-03T16:29:30.5386480Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5386807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5386917Z layer_outputs = layer_module( 2025-11-03T16:29:30.5387130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5387211Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5387472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5387549Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5387799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5387869Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5388143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5388256Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5388518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5388601Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5388604Z 2025-11-03T16:29:30.5388697Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5389026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5389091Z layer_outputs = layer_module( 2025-11-03T16:29:30.5389305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5389376Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5389667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5389751Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5390008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5390080Z self_outputs = self.self( 2025-11-03T16:29:30.5390337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5390422Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5390425Z 2025-11-03T16:29:30.5390518Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5390841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5390917Z layer_outputs = layer_module( 2025-11-03T16:29:30.5391122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5391204Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5391461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5391532Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5391796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5391862Z self_outputs = self.self( 2025-11-03T16:29:30.5392124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5392219Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5392541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5392744Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5392748Z 2025-11-03T16:29:30.5392841Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5393170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5393238Z layer_outputs = layer_module( 2025-11-03T16:29:30.5393449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5393521Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5393787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5393858Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5394201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5394285Z self_outputs = self.self( 2025-11-03T16:29:30.5394553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5394634Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5394638Z 2025-11-03T16:29:30.5394734Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5395117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5395189Z layer_outputs = layer_module( 2025-11-03T16:29:30.5395445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5395520Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5395780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5395864Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5396123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5396194Z self_outputs = self.self( 2025-11-03T16:29:30.5396452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5396546Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5396878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5397054Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5397057Z 2025-11-03T16:29:30.5397164Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5397496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5397568Z layer_outputs = layer_module( 2025-11-03T16:29:30.5397780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5397855Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5398130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5398200Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5398474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5398540Z self_outputs = self.self( 2025-11-03T16:29:30.5398840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5398935Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5399256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5399435Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5399438Z 2025-11-03T16:29:30.5399534Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5399874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5399941Z layer_outputs = layer_module( 2025-11-03T16:29:30.5400156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5400232Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5400497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5400573Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5400839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5400911Z self_outputs = self.self( 2025-11-03T16:29:30.5401175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5401268Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5401624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5401799Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5401802Z 2025-11-03T16:29:30.5401885Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5401959Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5402037Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5402107Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5402202Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5402542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5402608Z layer_outputs = layer_module( 2025-11-03T16:29:30.5402825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5402898Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5403166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5403241Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5403504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5403575Z self_outputs = self.self( 2025-11-03T16:29:30.5403841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5403949Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5404270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5404408Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5404757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5404850Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5404854Z 2025-11-03T16:29:30.5404933Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5405030Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5405366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5405441Z layer_outputs = layer_module( 2025-11-03T16:29:30.5405652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5405737Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5406012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5406092Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5406365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5406432Z self_outputs = self.self( 2025-11-03T16:29:30.5406710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5406780Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5406783Z 2025-11-03T16:29:30.5406887Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5407269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5407344Z layer_outputs = layer_module( 2025-11-03T16:29:30.5407554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5407632Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5407904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5407974Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5408247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5408312Z self_outputs = self.self( 2025-11-03T16:29:30.5408579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5408662Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5408667Z 2025-11-03T16:29:30.5408762Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5409116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5409184Z layer_outputs = layer_module( 2025-11-03T16:29:30.5409398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5409470Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5409732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5409805Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5410068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5410138Z self_outputs = self.self( 2025-11-03T16:29:30.5410393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5410504Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5410508Z 2025-11-03T16:29:30.5410607Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5410932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5411004Z layer_outputs = layer_module( 2025-11-03T16:29:30.5411210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5411288Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5411551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5411619Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5411885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5411951Z self_outputs = self.self( 2025-11-03T16:29:30.5412218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5412326Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5412655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5412823Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5413009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5413135Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5413139Z 2025-11-03T16:29:30.5413405Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5413751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5413818Z layer_outputs = layer_module( 2025-11-03T16:29:30.5414022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5414104Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5414365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5414442Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5414703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5414777Z self_outputs = self.self( 2025-11-03T16:29:30.5415034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5415146Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5415477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5415608Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5415907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5415992Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5416175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5416276Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5416279Z 2025-11-03T16:29:30.5416442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5416773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5416838Z layer_outputs = layer_module( 2025-11-03T16:29:30.5417050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5417122Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5417378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5417453Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5417709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5417783Z self_outputs = self.self( 2025-11-03T16:29:30.5418040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5418152Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5418472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5418611Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5418615Z 2025-11-03T16:29:30.5418715Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5419078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5419152Z layer_outputs = layer_module( 2025-11-03T16:29:30.5419356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5419436Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5419699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5419768Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5420032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5420095Z self_outputs = self.self( 2025-11-03T16:29:30.5420358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5420464Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5420791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5420940Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5420943Z 2025-11-03T16:29:30.5421037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5421367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5421432Z layer_outputs = layer_module( 2025-11-03T16:29:30.5421641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5421715Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5421975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5422050Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5422309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5422411Z self_outputs = self.self( 2025-11-03T16:29:30.5422669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5422849Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5422852Z 2025-11-03T16:29:30.5422946Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5423267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5423340Z layer_outputs = layer_module( 2025-11-03T16:29:30.5423545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5423624Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5423884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5423954Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5424217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5424319Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5424584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5424663Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5424666Z 2025-11-03T16:29:30.5424792Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5425114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5425182Z layer_outputs = layer_module( 2025-11-03T16:29:30.5425395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5425466Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5425732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5425810Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5426059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5426130Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5426392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5426504Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5426765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5426847Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5426850Z 2025-11-03T16:29:30.5426942Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5427262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5427332Z layer_outputs = layer_module( 2025-11-03T16:29:30.5427536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5427616Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5427875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5428011Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5428255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5428325Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5428602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5428702Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5428970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5429075Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5429280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5429353Z return self.act(input) 2025-11-03T16:29:30.5429358Z 2025-11-03T16:29:30.5429454Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5429785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5429851Z layer_outputs = layer_module( 2025-11-03T16:29:30.5430061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5430132Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5430392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5430476Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5430753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5430835Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5431094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5431207Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5431469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5431544Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5431547Z 2025-11-03T16:29:30.5431647Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5431972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5432043Z layer_outputs = layer_module( 2025-11-03T16:29:30.5432245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5432319Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5432586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5432656Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5432921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5432985Z self_outputs = self.self( 2025-11-03T16:29:30.5433248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5433323Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5433329Z 2025-11-03T16:29:30.5433421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5433751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5433846Z layer_outputs = layer_module( 2025-11-03T16:29:30.5434158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5434238Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5434506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5434586Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5434883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5434961Z self_outputs = self.self( 2025-11-03T16:29:30.5435219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5435325Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5435641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5435812Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5435815Z 2025-11-03T16:29:30.5435917Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5436242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5436317Z layer_outputs = layer_module( 2025-11-03T16:29:30.5436553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5436632Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5436893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5436961Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5437226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5437289Z self_outputs = self.self( 2025-11-03T16:29:30.5437552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5437625Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5437628Z 2025-11-03T16:29:30.5437721Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5438056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5438122Z layer_outputs = layer_module( 2025-11-03T16:29:30.5438333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5438403Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5438678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5438746Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5439002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5439072Z self_outputs = self.self( 2025-11-03T16:29:30.5439332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5439433Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5439746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5439952Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5439955Z 2025-11-03T16:29:30.5440049Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5440377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5440449Z layer_outputs = layer_module( 2025-11-03T16:29:30.5440654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5440733Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5440998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5441068Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5441336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5441398Z self_outputs = self.self( 2025-11-03T16:29:30.5441668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5441761Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5442083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5442275Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5442280Z 2025-11-03T16:29:30.5442375Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5442712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5442777Z layer_outputs = layer_module( 2025-11-03T16:29:30.5442992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5443064Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5443335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5443405Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5443675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5443746Z self_outputs = self.self( 2025-11-03T16:29:30.5444012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5444115Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5444440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5444613Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5444616Z 2025-11-03T16:29:30.5444693Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5444767Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5444845Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5444915Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5445016Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5445356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5445451Z layer_outputs = layer_module( 2025-11-03T16:29:30.5445665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5445737Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5446005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5446073Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5446329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5446397Z self_outputs = self.self( 2025-11-03T16:29:30.5446657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5446766Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5447079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5447220Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5447514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5447604Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5447608Z 2025-11-03T16:29:30.5447687Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5447779Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5448138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5448206Z layer_outputs = layer_module( 2025-11-03T16:29:30.5448417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5448489Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5448746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5448822Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5449080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5449148Z self_outputs = self.self( 2025-11-03T16:29:30.5449407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5449479Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5449482Z 2025-11-03T16:29:30.5449583Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5449907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5449975Z layer_outputs = layer_module( 2025-11-03T16:29:30.5450177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5450255Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5450512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5450580Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5450844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5450906Z self_outputs = self.self( 2025-11-03T16:29:30.5451168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5451278Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5451282Z 2025-11-03T16:29:30.5451377Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5451708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5451772Z layer_outputs = layer_module( 2025-11-03T16:29:30.5451980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5452052Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5452317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5452385Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5452643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5452717Z self_outputs = self.self( 2025-11-03T16:29:30.5452971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5453055Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5453058Z 2025-11-03T16:29:30.5453149Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5453472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5453544Z layer_outputs = layer_module( 2025-11-03T16:29:30.5453776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5453856Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5454116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5454191Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5454446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5454510Z self_outputs = self.self( 2025-11-03T16:29:30.5454773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5454880Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5455212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5455370Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5455560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5455651Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5455655Z 2025-11-03T16:29:30.5455746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5456081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5456146Z layer_outputs = layer_module( 2025-11-03T16:29:30.5456354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5456423Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5456683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5456790Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5457051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5457123Z self_outputs = self.self( 2025-11-03T16:29:30.5457381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5457497Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5457822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5457948Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5458251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5458337Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5458522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5458611Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5458614Z 2025-11-03T16:29:30.5458713Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5459033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5459100Z layer_outputs = layer_module( 2025-11-03T16:29:30.5459312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5459414Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5459681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5459751Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5460009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5460082Z self_outputs = self.self( 2025-11-03T16:29:30.5460339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5460451Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5460774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5460926Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5460929Z 2025-11-03T16:29:30.5461022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5461347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5461419Z layer_outputs = layer_module( 2025-11-03T16:29:30.5461621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5461700Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5461957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5462033Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5462290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5462355Z self_outputs = self.self( 2025-11-03T16:29:30.5462619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5462752Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5463083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5463225Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5463229Z 2025-11-03T16:29:30.5463328Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5463656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5463721Z layer_outputs = layer_module( 2025-11-03T16:29:30.5463934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5464007Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5464276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5464344Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5464603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5464674Z self_outputs = self.self( 2025-11-03T16:29:30.5464928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5465107Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5465111Z 2025-11-03T16:29:30.5465229Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5465563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5465633Z layer_outputs = layer_module( 2025-11-03T16:29:30.5465839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5465918Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5466178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5466252Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5466510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5466612Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5466878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5466957Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5466960Z 2025-11-03T16:29:30.5467060Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5467384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5467457Z layer_outputs = layer_module( 2025-11-03T16:29:30.5467661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5467734Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5468000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5468080Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5468331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5468438Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5468707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5468809Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5469069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5469152Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5469155Z 2025-11-03T16:29:30.5469250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5469584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5469650Z layer_outputs = layer_module( 2025-11-03T16:29:30.5469855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5469934Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5470195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5470277Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5470520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5470599Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5470862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5470988Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5471260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5471367Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5471576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5471641Z return self.act(input) 2025-11-03T16:29:30.5471645Z 2025-11-03T16:29:30.5471745Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5472074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5472140Z layer_outputs = layer_module( 2025-11-03T16:29:30.5472356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5472429Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5472698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5472778Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5473022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5473101Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5473364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5473485Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5473751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5473837Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5473840Z 2025-11-03T16:29:30.5473937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5474400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5474479Z layer_outputs = layer_module( 2025-11-03T16:29:30.5474692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5474775Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5475041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5475112Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5475387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5475455Z self_outputs = self.self( 2025-11-03T16:29:30.5475725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5475804Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5475807Z 2025-11-03T16:29:30.5475912Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5476243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5476311Z layer_outputs = layer_module( 2025-11-03T16:29:30.5476528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5476602Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5476905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5476977Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5477249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5477318Z self_outputs = self.self( 2025-11-03T16:29:30.5477583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5477686Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5478010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5478191Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5478194Z 2025-11-03T16:29:30.5478290Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5478631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5478702Z layer_outputs = layer_module( 2025-11-03T16:29:30.5478910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5478988Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5479253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5479329Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5479593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5479658Z self_outputs = self.self( 2025-11-03T16:29:30.5479931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5480005Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5480043Z 2025-11-03T16:29:30.5480148Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5480479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5480552Z layer_outputs = layer_module( 2025-11-03T16:29:30.5480757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5480829Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5481102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5481171Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5481446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5481514Z self_outputs = self.self( 2025-11-03T16:29:30.5481778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5481880Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5482201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5482384Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5482387Z 2025-11-03T16:29:30.5482482Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5482854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5482927Z layer_outputs = layer_module( 2025-11-03T16:29:30.5483136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5483221Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5483490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5483567Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5483833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5483904Z self_outputs = self.self( 2025-11-03T16:29:30.5484170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5484268Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5484600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5484775Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5484778Z 2025-11-03T16:29:30.5484883Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5485216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5485291Z layer_outputs = layer_module( 2025-11-03T16:29:30.5485500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5485575Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5485851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5485921Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5486448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5486515Z self_outputs = self.self( 2025-11-03T16:29:30.5486781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5486888Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5487209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5487387Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5487390Z 2025-11-03T16:29:30.5487470Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5487555Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5487628Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5487704Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5487810Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5488148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5488226Z layer_outputs = layer_module( 2025-11-03T16:29:30.5488440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5488516Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5488795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5488897Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5489174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5489243Z self_outputs = self.self( 2025-11-03T16:29:30.5489510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5489622Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5489949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5490094Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5490403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5490503Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5490506Z 2025-11-03T16:29:30.5490581Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5490679Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5491022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5491092Z layer_outputs = layer_module( 2025-11-03T16:29:30.5491317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5491392Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5491665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5491737Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5492004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5492079Z self_outputs = self.self( 2025-11-03T16:29:30.5492376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5492456Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5492460Z 2025-11-03T16:29:30.5492557Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5492898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5492972Z layer_outputs = layer_module( 2025-11-03T16:29:30.5493176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5493257Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5493518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5493592Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5493853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5493916Z self_outputs = self.self( 2025-11-03T16:29:30.5494185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5494258Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5494261Z 2025-11-03T16:29:30.5494363Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5494685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5494757Z layer_outputs = layer_module( 2025-11-03T16:29:30.5494996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5495073Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5495337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5495405Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5495666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5495729Z self_outputs = self.self( 2025-11-03T16:29:30.5495983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5496068Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5496072Z 2025-11-03T16:29:30.5496168Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5496498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5496566Z layer_outputs = layer_module( 2025-11-03T16:29:30.5496787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5496858Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5497117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5497192Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5497448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5497516Z self_outputs = self.self( 2025-11-03T16:29:30.5497778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5497888Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5498257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5498417Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5498604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5498697Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5498700Z 2025-11-03T16:29:30.5498800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5499127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5499193Z layer_outputs = layer_module( 2025-11-03T16:29:30.5499404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5499480Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5499744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5499812Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5500074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5500137Z self_outputs = self.self( 2025-11-03T16:29:30.5500393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5500507Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5500891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5501032Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5501325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5501410Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5501596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5501710Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5501713Z 2025-11-03T16:29:30.5501817Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5502143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5502217Z layer_outputs = layer_module( 2025-11-03T16:29:30.5502425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5502501Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5502769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5502838Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5503104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5503167Z self_outputs = self.self( 2025-11-03T16:29:30.5503428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5503546Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5503877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5504059Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5504063Z 2025-11-03T16:29:30.5504156Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5504489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5504555Z layer_outputs = layer_module( 2025-11-03T16:29:30.5504760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5504839Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5505103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5505180Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5505443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5505514Z self_outputs = self.self( 2025-11-03T16:29:30.5505774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5505882Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5506213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5506351Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5506355Z 2025-11-03T16:29:30.5506487Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5506814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5506889Z layer_outputs = layer_module( 2025-11-03T16:29:30.5507100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5507171Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5507446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5507514Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5507790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5507855Z self_outputs = self.self( 2025-11-03T16:29:30.5508123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5508310Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5508314Z 2025-11-03T16:29:30.5508407Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5508750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5508816Z layer_outputs = layer_module( 2025-11-03T16:29:30.5509030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5509103Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5509370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5509449Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5509719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5509858Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5510116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5510201Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5510204Z 2025-11-03T16:29:30.5510296Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5510618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5510692Z layer_outputs = layer_module( 2025-11-03T16:29:30.5510896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5510973Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5511231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5511308Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5511559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5511630Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5511898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5511999Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5512292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5512370Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5512374Z 2025-11-03T16:29:30.5512468Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5512798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5512863Z layer_outputs = layer_module( 2025-11-03T16:29:30.5513077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5513149Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5513610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5513696Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5513983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5514095Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5514377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5514490Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5514768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5514885Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5515092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5515159Z return self.act(input) 2025-11-03T16:29:30.5515162Z 2025-11-03T16:29:30.5515262Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5515595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5515670Z layer_outputs = layer_module( 2025-11-03T16:29:30.5515936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5516010Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5516286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5516365Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5516622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5516696Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5516967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5517092Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5517358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5517447Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5517451Z 2025-11-03T16:29:30.5517549Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5517890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5517957Z layer_outputs = layer_module( 2025-11-03T16:29:30.5518170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5518253Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5518572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5518654Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5518921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5518995Z self_outputs = self.self( 2025-11-03T16:29:30.5519260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5519338Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5519341Z 2025-11-03T16:29:30.5519444Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5519783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5519856Z layer_outputs = layer_module( 2025-11-03T16:29:30.5520067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5520143Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5520416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5520488Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5520765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5520831Z self_outputs = self.self( 2025-11-03T16:29:30.5521111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5521209Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5521535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5521717Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5521774Z 2025-11-03T16:29:30.5521872Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5522211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5522279Z layer_outputs = layer_module( 2025-11-03T16:29:30.5522500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5522574Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5522846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5522928Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5523199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5523276Z self_outputs = self.self( 2025-11-03T16:29:30.5523546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5523620Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5523624Z 2025-11-03T16:29:30.5523727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5524066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5524140Z layer_outputs = layer_module( 2025-11-03T16:29:30.5524353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5524466Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5524735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5524808Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5525079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5525145Z self_outputs = self.self( 2025-11-03T16:29:30.5525422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5525519Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5525853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5526033Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5526037Z 2025-11-03T16:29:30.5526135Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5526474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5526542Z layer_outputs = layer_module( 2025-11-03T16:29:30.5526758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5526831Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5527094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5527170Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5527437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5527506Z self_outputs = self.self( 2025-11-03T16:29:30.5527825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5527953Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5528267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5528434Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5528437Z 2025-11-03T16:29:30.5528539Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5528864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5528937Z layer_outputs = layer_module( 2025-11-03T16:29:30.5529142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5529221Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5529481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5529548Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5529814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5529878Z self_outputs = self.self( 2025-11-03T16:29:30.5530144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5530236Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5530580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5530755Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5530758Z 2025-11-03T16:29:30.5530834Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5530914Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5530986Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5531064Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5531156Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5531483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5531556Z layer_outputs = layer_module( 2025-11-03T16:29:30.5531762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5531843Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5532105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5532177Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5532446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5532511Z self_outputs = self.self( 2025-11-03T16:29:30.5532777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5532882Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5533206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5533342Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5533641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5533772Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5533776Z 2025-11-03T16:29:30.5533849Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5533950Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5534277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5534344Z layer_outputs = layer_module( 2025-11-03T16:29:30.5534561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5534638Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5534911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5534983Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5535253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5535315Z self_outputs = self.self( 2025-11-03T16:29:30.5535574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5535651Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5535654Z 2025-11-03T16:29:30.5535749Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5536084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5536178Z layer_outputs = layer_module( 2025-11-03T16:29:30.5536388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5536462Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5536727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5536804Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5537059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5537128Z self_outputs = self.self( 2025-11-03T16:29:30.5537384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5537457Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5537461Z 2025-11-03T16:29:30.5537565Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5537889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5537966Z layer_outputs = layer_module( 2025-11-03T16:29:30.5538168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5538247Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5538502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5538570Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5538836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5538901Z self_outputs = self.self( 2025-11-03T16:29:30.5539167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5539283Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5539286Z 2025-11-03T16:29:30.5539379Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5539718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5539784Z layer_outputs = layer_module( 2025-11-03T16:29:30.5539999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5540071Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5540351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5540421Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5540680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5540755Z self_outputs = self.self( 2025-11-03T16:29:30.5541015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5541129Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5541458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5541622Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5541805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5541929Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5541933Z 2025-11-03T16:29:30.5542034Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5542364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5542435Z layer_outputs = layer_module( 2025-11-03T16:29:30.5542640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5542712Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5542980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5543048Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5543317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5543383Z self_outputs = self.self( 2025-11-03T16:29:30.5543644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5543753Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5544079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5544212Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5544507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5544598Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5544779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5544871Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5544882Z 2025-11-03T16:29:30.5544977Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5545332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5545404Z layer_outputs = layer_module( 2025-11-03T16:29:30.5545612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5545689Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5545948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5546016Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5546285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5546350Z self_outputs = self.self( 2025-11-03T16:29:30.5546613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5546721Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5547052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5547193Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5547196Z 2025-11-03T16:29:30.5547290Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5547624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5547719Z layer_outputs = layer_module( 2025-11-03T16:29:30.5547935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5548010Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5548278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5548347Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5548605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5548678Z self_outputs = self.self( 2025-11-03T16:29:30.5548940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5549053Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5549378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5549518Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5549529Z 2025-11-03T16:29:30.5549622Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5549948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5550023Z layer_outputs = layer_module( 2025-11-03T16:29:30.5550232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5550311Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5550569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5550639Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5550908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5551002Z self_outputs = self.self( 2025-11-03T16:29:30.5551262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5551433Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5551437Z 2025-11-03T16:29:30.5551536Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5551858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5551923Z layer_outputs = layer_module( 2025-11-03T16:29:30.5552138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5552209Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5552478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5552545Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5552802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5552913Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5553169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5553255Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5553258Z 2025-11-03T16:29:30.5553351Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5553712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5553783Z layer_outputs = layer_module( 2025-11-03T16:29:30.5554058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5554148Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5554414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5554503Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5554754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5554838Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5555110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5555216Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5555505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5555582Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5555586Z 2025-11-03T16:29:30.5555688Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5556012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5556080Z layer_outputs = layer_module( 2025-11-03T16:29:30.5556299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5556372Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5556646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5556805Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5557061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5557134Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5557403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5557514Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5557782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5557896Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5558102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5558170Z return self.act(input) 2025-11-03T16:29:30.5558181Z 2025-11-03T16:29:30.5558279Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5558613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5558692Z layer_outputs = layer_module( 2025-11-03T16:29:30.5558907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5558989Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5559259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5559338Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5559628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5559704Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5559983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5560099Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5560372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5560449Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5560453Z 2025-11-03T16:29:30.5560549Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5560893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5560964Z layer_outputs = layer_module( 2025-11-03T16:29:30.5561186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5561261Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5561530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5561607Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5561881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5561955Z self_outputs = self.self( 2025-11-03T16:29:30.5562222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5562308Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5562311Z 2025-11-03T16:29:30.5562410Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5562746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5562861Z layer_outputs = layer_module( 2025-11-03T16:29:30.5563078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5563158Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5563431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5563504Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5563782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5563848Z self_outputs = self.self( 2025-11-03T16:29:30.5564132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5564234Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5564573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5564749Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5564753Z 2025-11-03T16:29:30.5564851Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5565202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5565270Z layer_outputs = layer_module( 2025-11-03T16:29:30.5565522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5565598Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5565873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5565948Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5566218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5566290Z self_outputs = self.self( 2025-11-03T16:29:30.5566558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5566638Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5566642Z 2025-11-03T16:29:30.5566738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5567085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5567151Z layer_outputs = layer_module( 2025-11-03T16:29:30.5567363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5567445Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5567712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5567788Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5568054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5568119Z self_outputs = self.self( 2025-11-03T16:29:30.5568393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5568505Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5568830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5569033Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5569037Z 2025-11-03T16:29:30.5569141Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5569473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5569540Z layer_outputs = layer_module( 2025-11-03T16:29:30.5569752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5569824Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5570097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5570169Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5570432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5570503Z self_outputs = self.self( 2025-11-03T16:29:30.5570763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5570864Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5571182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5571354Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5571392Z 2025-11-03T16:29:30.5571488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5571812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5571888Z layer_outputs = layer_module( 2025-11-03T16:29:30.5572091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5572171Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5572433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5572507Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5572767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5572833Z self_outputs = self.self( 2025-11-03T16:29:30.5573098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5573193Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5573514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5573679Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5573682Z 2025-11-03T16:29:30.5573763Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5573835Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5573906Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5573987Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5574081Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5574417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5574515Z layer_outputs = layer_module( 2025-11-03T16:29:30.5574814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5574892Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5575152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5575228Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5575485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5575547Z self_outputs = self.self( 2025-11-03T16:29:30.5575811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5575914Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5576237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5576373Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5576678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5576769Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5576772Z 2025-11-03T16:29:30.5576844Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5576947Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5577301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5577380Z layer_outputs = layer_module( 2025-11-03T16:29:30.5577584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5577666Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5577924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5577994Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5578258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5578322Z self_outputs = self.self( 2025-11-03T16:29:30.5578588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5578660Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5578663Z 2025-11-03T16:29:30.5578759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5579091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5579161Z layer_outputs = layer_module( 2025-11-03T16:29:30.5579376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5579451Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5579718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5579786Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5580042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5580117Z self_outputs = self.self( 2025-11-03T16:29:30.5580372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5580483Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5580486Z 2025-11-03T16:29:30.5580581Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5580911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5580979Z layer_outputs = layer_module( 2025-11-03T16:29:30.5581184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5581264Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5581534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5581615Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5581873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5581939Z self_outputs = self.self( 2025-11-03T16:29:30.5582208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5582287Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5582290Z 2025-11-03T16:29:30.5582394Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5582725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5582798Z layer_outputs = layer_module( 2025-11-03T16:29:30.5583046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5583122Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5583397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5583471Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5583746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5583811Z self_outputs = self.self( 2025-11-03T16:29:30.5584075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5584195Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5584536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5584710Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5584902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5585014Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5585017Z 2025-11-03T16:29:30.5585110Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5585439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5585515Z layer_outputs = layer_module( 2025-11-03T16:29:30.5585726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5585808Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5586078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5586155Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5586452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5586517Z self_outputs = self.self( 2025-11-03T16:29:30.5586787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5586897Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5587239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5587369Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5587672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5587765Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5587948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5588047Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5588050Z 2025-11-03T16:29:30.5588146Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5588488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5588556Z layer_outputs = layer_module( 2025-11-03T16:29:30.5588764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5588843Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5589135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5589217Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5589479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5589549Z self_outputs = self.self( 2025-11-03T16:29:30.5589818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5589925Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5590264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5590407Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5590413Z 2025-11-03T16:29:30.5590516Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5590857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5590937Z layer_outputs = layer_module( 2025-11-03T16:29:30.5591145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5591220Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5591491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5591563Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5591833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5591902Z self_outputs = self.self( 2025-11-03T16:29:30.5592167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5592319Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5592657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5592812Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5592816Z 2025-11-03T16:29:30.5592913Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5593257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5593325Z layer_outputs = layer_module( 2025-11-03T16:29:30.5593538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5593621Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5593896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5594028Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5594304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5594371Z self_outputs = self.self( 2025-11-03T16:29:30.5594645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5594824Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5594828Z 2025-11-03T16:29:30.5594970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5595311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5595393Z layer_outputs = layer_module( 2025-11-03T16:29:30.5595601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5595676Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5595957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5596030Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5596305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5596413Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5596691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5596772Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5596778Z 2025-11-03T16:29:30.5596875Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5597215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5597284Z layer_outputs = layer_module( 2025-11-03T16:29:30.5597501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5597575Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5597862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5597946Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5598198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5598315Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5598589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5598699Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5598959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5599039Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5599042Z 2025-11-03T16:29:30.5599148Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5599482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5599560Z layer_outputs = layer_module( 2025-11-03T16:29:30.5599765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5599850Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5600108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5600188Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5600441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5600516Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5600786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5600918Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5601174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5601288Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5601486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5601557Z return self.act(input) 2025-11-03T16:29:30.5601561Z 2025-11-03T16:29:30.5601653Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5601983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5602051Z layer_outputs = layer_module( 2025-11-03T16:29:30.5602251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5602332Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5602587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5602673Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5602911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5602981Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5603244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5603356Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5603618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5603695Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5603701Z 2025-11-03T16:29:30.5603801Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5604122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5604219Z layer_outputs = layer_module( 2025-11-03T16:29:30.5604430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5604499Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5604760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5604830Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5605089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5605156Z self_outputs = self.self( 2025-11-03T16:29:30.5605411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-11-03T16:29:30.5605496Z query_vectors = self.query(hidden_states) 2025-11-03T16:29:30.5605499Z 2025-11-03T16:29:30.5605591Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5605918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5605986Z layer_outputs = layer_module( 2025-11-03T16:29:30.5606187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5606264Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5606563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5606642Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5606902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5606977Z self_outputs = self.self( 2025-11-03T16:29:30.5607236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5607331Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5607655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5607824Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5607827Z 2025-11-03T16:29:30.5607927Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5608261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5608336Z layer_outputs = layer_module( 2025-11-03T16:29:30.5608542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5608613Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5608883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5608952Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5609222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5609286Z self_outputs = self.self( 2025-11-03T16:29:30.5609548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-11-03T16:29:30.5609628Z key_vectors = self.key(hidden_states) 2025-11-03T16:29:30.5609631Z 2025-11-03T16:29:30.5609758Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5610090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5610159Z layer_outputs = layer_module( 2025-11-03T16:29:30.5610370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5610443Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5610705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5610783Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5611044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5611117Z self_outputs = self.self( 2025-11-03T16:29:30.5611378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5611472Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5611797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5611965Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5611968Z 2025-11-03T16:29:30.5612071Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5612443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5612520Z layer_outputs = layer_module( 2025-11-03T16:29:30.5612724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5612800Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5613066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5613136Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5613552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5613625Z self_outputs = self.self( 2025-11-03T16:29:30.5613895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5613989Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5614308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5614488Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5614492Z 2025-11-03T16:29:30.5614587Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5614921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5614987Z layer_outputs = layer_module( 2025-11-03T16:29:30.5615198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5615271Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5615532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5615609Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5615868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5615995Z self_outputs = self.self( 2025-11-03T16:29:30.5616256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-11-03T16:29:30.5616350Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5616672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5616837Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-11-03T16:29:30.5616841Z 2025-11-03T16:29:30.5616923Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5617000Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5617079Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5617153Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5617247Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5617583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5617649Z layer_outputs = layer_module( 2025-11-03T16:29:30.5617863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5617935Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5618194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5618271Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5618568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5618645Z self_outputs = self.self( 2025-11-03T16:29:30.5618900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-11-03T16:29:30.5619010Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-11-03T16:29:30.5619322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-11-03T16:29:30.5619453Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-11-03T16:29:30.5619760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 764, in _mask_invalid_locations 2025-11-03T16:29:30.5619848Z ).where(beginning_mask.bool(), beginning_input) 2025-11-03T16:29:30.5619854Z 2025-11-03T16:29:30.5619933Z cudagraph partition due to non gpu ops 2025-11-03T16:29:30.5620025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5620355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5620421Z layer_outputs = layer_module( 2025-11-03T16:29:30.5620622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5620699Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5620958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5621034Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5621293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5621356Z self_outputs = self.self( 2025-11-03T16:29:30.5621618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-11-03T16:29:30.5621716Z attn_scores += diagonal_mask 2025-11-03T16:29:30.5621719Z 2025-11-03T16:29:30.5621822Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5622159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5622230Z layer_outputs = layer_module( 2025-11-03T16:29:30.5622438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5622510Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5622787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5622857Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5623133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5623196Z self_outputs = self.self( 2025-11-03T16:29:30.5623460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-11-03T16:29:30.5623541Z attn_probs = nn.functional.softmax( 2025-11-03T16:29:30.5623545Z 2025-11-03T16:29:30.5623641Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5623979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5624044Z layer_outputs = layer_module( 2025-11-03T16:29:30.5624290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5624364Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5624625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5624703Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5624963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5625035Z self_outputs = self.self( 2025-11-03T16:29:30.5625292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-11-03T16:29:30.5625371Z value_vectors = self.value(hidden_states) 2025-11-03T16:29:30.5625381Z 2025-11-03T16:29:30.5625474Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5625796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5625873Z layer_outputs = layer_module( 2025-11-03T16:29:30.5626077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5626156Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5626413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5626482Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5626761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5626824Z self_outputs = self.self( 2025-11-03T16:29:30.5627090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5627201Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5627577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5627740Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-11-03T16:29:30.5627922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5628023Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5628026Z 2025-11-03T16:29:30.5628121Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5628451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5628518Z layer_outputs = layer_module( 2025-11-03T16:29:30.5628723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5628806Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5629072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5629151Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5629417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5629488Z self_outputs = self.self( 2025-11-03T16:29:30.5629754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5629864Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5630235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5630369Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-11-03T16:29:30.5630680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-11-03T16:29:30.5630767Z chunked_hidden_states = nn.functional.pad( 2025-11-03T16:29:30.5630961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5413, in pad 2025-11-03T16:29:30.5631054Z return torch._C._nn.pad(input, pad, mode, value) 2025-11-03T16:29:30.5631058Z 2025-11-03T16:29:30.5631157Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5631503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5631570Z layer_outputs = layer_module( 2025-11-03T16:29:30.5631791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5631869Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5632135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5632214Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5632481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5632553Z self_outputs = self.self( 2025-11-03T16:29:30.5632820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5632937Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5633274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5633451Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5633455Z 2025-11-03T16:29:30.5633557Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5633893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5634017Z layer_outputs = layer_module( 2025-11-03T16:29:30.5634239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5634320Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5634593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5634665Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5634949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5635019Z self_outputs = self.self( 2025-11-03T16:29:30.5635307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-11-03T16:29:30.5635416Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-11-03T16:29:30.5635751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-11-03T16:29:30.5635904Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-11-03T16:29:30.5635908Z 2025-11-03T16:29:30.5636036Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5636379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5636452Z layer_outputs = layer_module( 2025-11-03T16:29:30.5636672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5636747Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5637014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5637094Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5637358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-11-03T16:29:30.5637429Z self_outputs = self.self( 2025-11-03T16:29:30.5637693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-11-03T16:29:30.5637876Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-11-03T16:29:30.5637882Z 2025-11-03T16:29:30.5637978Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5638312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5638387Z layer_outputs = layer_module( 2025-11-03T16:29:30.5638594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5638674Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5638939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-11-03T16:29:30.5639012Z self_attn_outputs = self.attention( 2025-11-03T16:29:30.5639284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-11-03T16:29:30.5639422Z attn_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:29:30.5639699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-11-03T16:29:30.5639779Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5639782Z 2025-11-03T16:29:30.5639882Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5640221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5640290Z layer_outputs = layer_module( 2025-11-03T16:29:30.5640514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5640588Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5640863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5640947Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5641209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5641282Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5641556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5641669Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5641940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-11-03T16:29:30.5642058Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5642062Z 2025-11-03T16:29:30.5642163Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5642502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5642578Z layer_outputs = layer_module( 2025-11-03T16:29:30.5642788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5642870Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5643139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5643225Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5643474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5643547Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5643827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-11-03T16:29:30.5643933Z intermediate_output = self.intermediate(attn_output) 2025-11-03T16:29:30.5644204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-11-03T16:29:30.5644311Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:29:30.5644522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:29:30.5644597Z return self.act(input) 2025-11-03T16:29:30.5644600Z 2025-11-03T16:29:30.5644696Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:29:30.5645039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-11-03T16:29:30.5645107Z layer_outputs = layer_module( 2025-11-03T16:29:30.5645356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:29:30.5645430Z return super().__call__(*args, **kwargs) 2025-11-03T16:29:30.5645701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-11-03T16:29:30.5645785Z layer_output = apply_chunking_to_forward( 2025-11-03T16:29:30.5646033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:29:30.5646108Z return forward_fn(*input_tensors) 2025-11-03T16:29:30.5646381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-11-03T16:29:30.5646504Z layer_output = self.output(intermediate_output, attn_output) 2025-11-03T16:29:30.5646769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-11-03T16:29:30.5646848Z hidden_states = self.dense(hidden_states) 2025-11-03T16:29:30.5646851Z 2025-11-03T16:30:34.9752990Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:34.9756329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-11-03T16:30:34.9760763Z prediction_scores = self.lm_head(sequence_output) 2025-11-03T16:30:34.9764761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward 2025-11-03T16:30:34.9769881Z x = self.dense(features) 2025-11-03T16:30:34.9774019Z 2025-11-03T16:30:34.9779717Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:34.9785108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-11-03T16:30:34.9787110Z prediction_scores = self.lm_head(sequence_output) 2025-11-03T16:30:34.9787784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward 2025-11-03T16:30:34.9788226Z x = self.decoder(x) 2025-11-03T16:30:34.9788344Z 2025-11-03T16:30:34.9788456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:34.9788983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1723, in torch_dynamo_resume_in_forward_at_1703 2025-11-03T16:30:34.9792378Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:30:34.9792661Z 2025-11-03T16:30:36.2288987Z Compilation time (from dynamo_timed): 96.287989813 2025-11-03T16:30:36.2541964Z pass 2025-11-03T16:30:36.2546421Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:30:36.2548913Z TIMING: gc:0.00539 entire_frame_compile:96.28799 _recursive_pre_grad_passes:0.0188 _recursive_joint_graph_passes:0.93713 _recursive_post_grad_passes:1.58503 async_compile.wait:2.72077 code_gen:73.92896 inductor_compile:80.96278 backend_compile:90.87579 total_wall_time:96.28799 2025-11-03T16:30:36.2549892Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:39806 | FakeTensor.__torch_dispatch__:16210 | ProxyTorchDispatchMode.__torch_dispatch__:9995 2025-11-03T16:30:36.2552614Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-11-03T16:30:39.6917657Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:30:39.6918480Z import pynvml # type: ignore[import] 2025-11-03T16:30:42.8631460Z 2025-11-03T16:30:45.0803853Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:30:45.0806214Z loading model: 0it [00:02, ?it/s] 2025-11-03T16:30:45.0824043Z cpu eval BartForCausalLM 2025-11-03T16:30:46.6411601Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:30:47.2110345Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:30:47.8319623Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:30:54.5067049Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5067587Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5067934Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5068310Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5068670Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5068958Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5069296Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5069587Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5069896Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5070194Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5070480Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5070775Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5071609Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5072049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5072385Z res = mod(**inputs) 2025-11-03T16:30:54.5072788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5073570Z outputs = self.model.decoder( 2025-11-03T16:30:54.5074100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5074518Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5074895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5075297Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5075719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5076109Z return func(*args, **kwargs) 2025-11-03T16:30:54.5076488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5076909Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5077332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5077718Z return func(*args, **kwargs) 2025-11-03T16:30:54.5078092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5078563Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5078779Z 2025-11-03T16:30:54.5078889Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5079262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5079592Z res = mod(**inputs) 2025-11-03T16:30:54.5079953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5080350Z outputs = self.model.decoder( 2025-11-03T16:30:54.5080738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5081145Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5081526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5082026Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5082445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5082867Z return func(*args, **kwargs) 2025-11-03T16:30:54.5083248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5083660Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5084071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5084447Z return func(*args, **kwargs) 2025-11-03T16:30:54.5084809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5085189Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5085325Z 2025-11-03T16:30:54.5085431Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5085783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5086095Z res = mod(**inputs) 2025-11-03T16:30:54.5086447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5086819Z outputs = self.model.decoder( 2025-11-03T16:30:54.5087178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5087548Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5088585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5088946Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5089303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5089660Z return func(*args, **kwargs) 2025-11-03T16:30:54.5090010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5090512Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5090893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5091236Z return func(*args, **kwargs) 2025-11-03T16:30:54.5091586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5091964Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5092100Z 2025-11-03T16:30:54.5092188Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5092418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5092760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5093069Z res = mod(**inputs) 2025-11-03T16:30:54.5093416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5093800Z outputs = self.model.decoder( 2025-11-03T16:30:54.5094148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5094523Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5094864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5095222Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5095592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5095995Z return func(*args, **kwargs) 2025-11-03T16:30:54.5096388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5096785Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5097177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5097535Z return func(*args, **kwargs) 2025-11-03T16:30:54.5097894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5098294Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5098735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5099211Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5099398Z 2025-11-03T16:30:54.5099501Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5099858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5100181Z res = mod(**inputs) 2025-11-03T16:30:54.5100530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5100902Z outputs = self.model.decoder( 2025-11-03T16:30:54.5101273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5101659Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5102000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5102395Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5102761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5103132Z return func(*args, **kwargs) 2025-11-03T16:30:54.5103492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5103890Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5104279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5104634Z return func(*args, **kwargs) 2025-11-03T16:30:54.5104994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5105378Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5105510Z 2025-11-03T16:30:54.5105622Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5105966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5106286Z res = mod(**inputs) 2025-11-03T16:30:54.5106638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5107023Z outputs = self.model.decoder( 2025-11-03T16:30:54.5107402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5107774Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5108120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5108478Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5108864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5109240Z return func(*args, **kwargs) 2025-11-03T16:30:54.5109606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5110118Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5110313Z 2025-11-03T16:30:54.5110423Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5110803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5111152Z res = mod(**inputs) 2025-11-03T16:30:54.5111536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5111954Z outputs = self.model.decoder( 2025-11-03T16:30:54.5112365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5112782Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5113165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5113798Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5114297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5114721Z return func(*args, **kwargs) 2025-11-03T16:30:54.5115114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5115591Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5116021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5116397Z return self.act(input) 2025-11-03T16:30:54.5116519Z 2025-11-03T16:30:54.5116770Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5117157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5117513Z res = mod(**inputs) 2025-11-03T16:30:54.5117904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5118328Z outputs = self.model.decoder( 2025-11-03T16:30:54.5118742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5119153Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5119543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5119938Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5120353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5120764Z return func(*args, **kwargs) 2025-11-03T16:30:54.5121169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5121604Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5121753Z 2025-11-03T16:30:54.5121878Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5122267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5122613Z res = mod(**inputs) 2025-11-03T16:30:54.5123004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5123421Z outputs = self.model.decoder( 2025-11-03T16:30:54.5123795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5124179Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5124525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5124892Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5125316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5125685Z return func(*args, **kwargs) 2025-11-03T16:30:54.5126045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5126468Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5126888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5127282Z return func(*args, **kwargs) 2025-11-03T16:30:54.5127653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5128110Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5128325Z 2025-11-03T16:30:54.5128430Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5128796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5129122Z res = mod(**inputs) 2025-11-03T16:30:54.5129480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5129862Z outputs = self.model.decoder( 2025-11-03T16:30:54.5130240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5130626Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5130975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5131370Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5131756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5132131Z return func(*args, **kwargs) 2025-11-03T16:30:54.5132503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5132908Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5133283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5133639Z return func(*args, **kwargs) 2025-11-03T16:30:54.5133990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5134371Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5134496Z 2025-11-03T16:30:54.5134605Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5134938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5135246Z res = mod(**inputs) 2025-11-03T16:30:54.5135581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5135946Z outputs = self.model.decoder( 2025-11-03T16:30:54.5136295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5136663Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5136996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5137342Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5137701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5138053Z return func(*args, **kwargs) 2025-11-03T16:30:54.5138404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5138825Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5139204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5139559Z return func(*args, **kwargs) 2025-11-03T16:30:54.5139905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5140285Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5140420Z 2025-11-03T16:30:54.5140509Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5140742Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5141075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5141389Z res = mod(**inputs) 2025-11-03T16:30:54.5141735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5142107Z outputs = self.model.decoder( 2025-11-03T16:30:54.5142468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5142828Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5143165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5143508Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5143869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5144217Z return func(*args, **kwargs) 2025-11-03T16:30:54.5144603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5144993Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5145377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5145734Z return func(*args, **kwargs) 2025-11-03T16:30:54.5146076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5146469Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5146899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5147361Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5147536Z 2025-11-03T16:30:54.5147644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5147984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5148293Z res = mod(**inputs) 2025-11-03T16:30:54.5148634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5149011Z outputs = self.model.decoder( 2025-11-03T16:30:54.5149367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5149738Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5150082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5150443Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5150818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5151175Z return func(*args, **kwargs) 2025-11-03T16:30:54.5151539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5151974Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5152369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5152736Z return func(*args, **kwargs) 2025-11-03T16:30:54.5153091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5153476Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5153617Z 2025-11-03T16:30:54.5153729Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5154202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5154545Z res = mod(**inputs) 2025-11-03T16:30:54.5154934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5155351Z outputs = self.model.decoder( 2025-11-03T16:30:54.5155744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5156122Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5156461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5156824Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5157199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5157568Z return func(*args, **kwargs) 2025-11-03T16:30:54.5157927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5158388Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5158567Z 2025-11-03T16:30:54.5158671Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5159027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5159343Z res = mod(**inputs) 2025-11-03T16:30:54.5159686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5160064Z outputs = self.model.decoder( 2025-11-03T16:30:54.5160431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5160829Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5161180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5161530Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5161906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5162271Z return func(*args, **kwargs) 2025-11-03T16:30:54.5162637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5163050Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5163425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5163767Z return self.act(input) 2025-11-03T16:30:54.5163875Z 2025-11-03T16:30:54.5163983Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5164332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5164640Z res = mod(**inputs) 2025-11-03T16:30:54.5164998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5165365Z outputs = self.model.decoder( 2025-11-03T16:30:54.5165724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5166124Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5166448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5166797Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5167160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5167512Z return func(*args, **kwargs) 2025-11-03T16:30:54.5167856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5168227Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5168365Z 2025-11-03T16:30:54.5168466Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5168809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5169116Z res = mod(**inputs) 2025-11-03T16:30:54.5169450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5169815Z outputs = self.model.decoder( 2025-11-03T16:30:54.5170169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5170535Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5170867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5171210Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5171620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5171976Z return func(*args, **kwargs) 2025-11-03T16:30:54.5172325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5172714Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5173108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5173473Z return func(*args, **kwargs) 2025-11-03T16:30:54.5173837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5174285Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5174485Z 2025-11-03T16:30:54.5174587Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5174938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5175253Z res = mod(**inputs) 2025-11-03T16:30:54.5175604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5175985Z outputs = self.model.decoder( 2025-11-03T16:30:54.5176348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5176725Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5177067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5177421Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5177782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5178149Z return func(*args, **kwargs) 2025-11-03T16:30:54.5178510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5178907Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5179332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5179689Z return func(*args, **kwargs) 2025-11-03T16:30:54.5180063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5180449Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5180582Z 2025-11-03T16:30:54.5180690Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5181039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5181347Z res = mod(**inputs) 2025-11-03T16:30:54.5181696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5182073Z outputs = self.model.decoder( 2025-11-03T16:30:54.5182436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5182817Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5183167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5183534Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5183905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5184269Z return func(*args, **kwargs) 2025-11-03T16:30:54.5184622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5185023Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5185450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5185824Z return func(*args, **kwargs) 2025-11-03T16:30:54.5186198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5186590Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5186736Z 2025-11-03T16:30:54.5186816Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5187052Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5187405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5187716Z res = mod(**inputs) 2025-11-03T16:30:54.5188071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5188453Z outputs = self.model.decoder( 2025-11-03T16:30:54.5188827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5189206Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5189551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5189914Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5190289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5190657Z return func(*args, **kwargs) 2025-11-03T16:30:54.5191015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5191417Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5191808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5192185Z return func(*args, **kwargs) 2025-11-03T16:30:54.5192538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5192956Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5193386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5193854Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5194112Z 2025-11-03T16:30:54.5194235Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5194603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5194937Z res = mod(**inputs) 2025-11-03T16:30:54.5195282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5195659Z outputs = self.model.decoder( 2025-11-03T16:30:54.5196020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5196384Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5196723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5197072Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5197442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5197801Z return func(*args, **kwargs) 2025-11-03T16:30:54.5198148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5198542Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5198963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5199323Z return func(*args, **kwargs) 2025-11-03T16:30:54.5199682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5200058Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5200195Z 2025-11-03T16:30:54.5200293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5200635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5200949Z res = mod(**inputs) 2025-11-03T16:30:54.5201285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5201655Z outputs = self.model.decoder( 2025-11-03T16:30:54.5202013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5202383Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5202720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5203065Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5203425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5203785Z return func(*args, **kwargs) 2025-11-03T16:30:54.5204140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5204552Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5204718Z 2025-11-03T16:30:54.5204816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5205160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5205471Z res = mod(**inputs) 2025-11-03T16:30:54.5205814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5206211Z outputs = self.model.decoder( 2025-11-03T16:30:54.5206584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5206946Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5207276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5207627Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5207987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5208337Z return func(*args, **kwargs) 2025-11-03T16:30:54.5208682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5209090Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5209455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5209778Z return self.act(input) 2025-11-03T16:30:54.5209889Z 2025-11-03T16:30:54.5209985Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5210320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5210623Z res = mod(**inputs) 2025-11-03T16:30:54.5210946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5211309Z outputs = self.model.decoder( 2025-11-03T16:30:54.5211663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5212054Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5212385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5212721Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5213076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5213624Z return func(*args, **kwargs) 2025-11-03T16:30:54.5213978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5214338Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5214474Z 2025-11-03T16:30:54.5214571Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5214907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5215219Z res = mod(**inputs) 2025-11-03T16:30:54.5215558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5215915Z outputs = self.model.decoder( 2025-11-03T16:30:54.5216284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5216646Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5216971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5217308Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5217650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5217995Z return func(*args, **kwargs) 2025-11-03T16:30:54.5218343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5218724Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5219095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5219521Z return func(*args, **kwargs) 2025-11-03T16:30:54.5219868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5220297Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5220493Z 2025-11-03T16:30:54.5220600Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5220930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5221233Z res = mod(**inputs) 2025-11-03T16:30:54.5221571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5221937Z outputs = self.model.decoder( 2025-11-03T16:30:54.5222296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5222651Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5222981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5223321Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5223674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5224015Z return func(*args, **kwargs) 2025-11-03T16:30:54.5224360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5224744Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5225169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5225524Z return func(*args, **kwargs) 2025-11-03T16:30:54.5225883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5226262Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5226398Z 2025-11-03T16:30:54.5226498Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5226839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5227151Z res = mod(**inputs) 2025-11-03T16:30:54.5227483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5227853Z outputs = self.model.decoder( 2025-11-03T16:30:54.5228209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5228578Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5228905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5229251Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5229608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5229964Z return func(*args, **kwargs) 2025-11-03T16:30:54.5230310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5230689Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5231068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5231419Z return func(*args, **kwargs) 2025-11-03T16:30:54.5231770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5232146Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5232281Z 2025-11-03T16:30:54.5232362Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5232623Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5232951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5233251Z res = mod(**inputs) 2025-11-03T16:30:54.5233576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5233991Z outputs = self.model.decoder( 2025-11-03T16:30:54.5234362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5234727Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5235068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5235416Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5235815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5236177Z return func(*args, **kwargs) 2025-11-03T16:30:54.5236528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5236910Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5237297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5237691Z return func(*args, **kwargs) 2025-11-03T16:30:54.5238045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5238436Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5238892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5239368Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5239552Z 2025-11-03T16:30:54.5239651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5239998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5240310Z res = mod(**inputs) 2025-11-03T16:30:54.5240648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5241023Z outputs = self.model.decoder( 2025-11-03T16:30:54.5241387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5241765Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5242100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5242452Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5242821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5243184Z return func(*args, **kwargs) 2025-11-03T16:30:54.5243539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5243927Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5244321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5244679Z return func(*args, **kwargs) 2025-11-03T16:30:54.5245033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5245419Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5245549Z 2025-11-03T16:30:54.5245648Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5246026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5246337Z res = mod(**inputs) 2025-11-03T16:30:54.5246678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5247039Z outputs = self.model.decoder( 2025-11-03T16:30:54.5247396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5247763Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5248100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5248445Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5248800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5249156Z return func(*args, **kwargs) 2025-11-03T16:30:54.5249507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5249956Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5250118Z 2025-11-03T16:30:54.5250224Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5250565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5250865Z res = mod(**inputs) 2025-11-03T16:30:54.5251202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5251568Z outputs = self.model.decoder( 2025-11-03T16:30:54.5251949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5252318Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5252654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5252999Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5253366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5253717Z return func(*args, **kwargs) 2025-11-03T16:30:54.5254071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5254479Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5254854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5255187Z return self.act(input) 2025-11-03T16:30:54.5255296Z 2025-11-03T16:30:54.5255396Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5255739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5256054Z res = mod(**inputs) 2025-11-03T16:30:54.5256399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5256766Z outputs = self.model.decoder( 2025-11-03T16:30:54.5257128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5257497Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5257832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5258178Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5258534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5258902Z return func(*args, **kwargs) 2025-11-03T16:30:54.5259245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5259644Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5259772Z 2025-11-03T16:30:54.5259874Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5260198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5260497Z res = mod(**inputs) 2025-11-03T16:30:54.5260828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5261185Z outputs = self.model.decoder( 2025-11-03T16:30:54.5261526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5261885Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5262209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5262550Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5262899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5263237Z return func(*args, **kwargs) 2025-11-03T16:30:54.5263580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5263965Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5264345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5264689Z return func(*args, **kwargs) 2025-11-03T16:30:54.5265077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5265520Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5265718Z 2025-11-03T16:30:54.5265826Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5266172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5266471Z res = mod(**inputs) 2025-11-03T16:30:54.5266814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5267182Z outputs = self.model.decoder( 2025-11-03T16:30:54.5267538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5267901Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5268233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5268583Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5268946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5269309Z return func(*args, **kwargs) 2025-11-03T16:30:54.5269653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5270045Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5270425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5270781Z return func(*args, **kwargs) 2025-11-03T16:30:54.5271136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5271504Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5271645Z 2025-11-03T16:30:54.5271746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5272087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5272446Z res = mod(**inputs) 2025-11-03T16:30:54.5272789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5273148Z outputs = self.model.decoder( 2025-11-03T16:30:54.5273507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5273874Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5274298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5274651Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5275035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5275401Z return func(*args, **kwargs) 2025-11-03T16:30:54.5275769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5276171Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5276557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5276927Z return func(*args, **kwargs) 2025-11-03T16:30:54.5277290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5277681Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5277822Z 2025-11-03T16:30:54.5277910Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5278142Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5278529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5278844Z res = mod(**inputs) 2025-11-03T16:30:54.5279195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5279565Z outputs = self.model.decoder( 2025-11-03T16:30:54.5279932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5280311Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5280648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5281001Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5281364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5281725Z return func(*args, **kwargs) 2025-11-03T16:30:54.5282087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5282487Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5282875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5283245Z return func(*args, **kwargs) 2025-11-03T16:30:54.5283619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5284039Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5284467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5284944Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5285132Z 2025-11-03T16:30:54.5285237Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5285590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5285938Z res = mod(**inputs) 2025-11-03T16:30:54.5286289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5286665Z outputs = self.model.decoder( 2025-11-03T16:30:54.5287034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5287414Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5287759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5288115Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5288483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5288850Z return func(*args, **kwargs) 2025-11-03T16:30:54.5289213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5289619Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5290003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5290369Z return func(*args, **kwargs) 2025-11-03T16:30:54.5290735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5291140Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5291272Z 2025-11-03T16:30:54.5291381Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5291728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5292090Z res = mod(**inputs) 2025-11-03T16:30:54.5292452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5292836Z outputs = self.model.decoder( 2025-11-03T16:30:54.5293204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5293589Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5293937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5294282Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5294638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5294982Z return func(*args, **kwargs) 2025-11-03T16:30:54.5295334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5295742Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5295905Z 2025-11-03T16:30:54.5296012Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5296343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5296640Z res = mod(**inputs) 2025-11-03T16:30:54.5296977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5297340Z outputs = self.model.decoder( 2025-11-03T16:30:54.5297700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5298055Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5298379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5298722Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5299076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5299451Z return func(*args, **kwargs) 2025-11-03T16:30:54.5299786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5300183Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5300541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5300859Z return self.act(input) 2025-11-03T16:30:54.5300964Z 2025-11-03T16:30:54.5301067Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5301395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5301700Z res = mod(**inputs) 2025-11-03T16:30:54.5302032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5302392Z outputs = self.model.decoder( 2025-11-03T16:30:54.5302738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5303099Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5303427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5303769Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5304119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5304459Z return func(*args, **kwargs) 2025-11-03T16:30:54.5304807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5305229Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5305360Z 2025-11-03T16:30:54.5305462Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5305800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5306096Z res = mod(**inputs) 2025-11-03T16:30:54.5306430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5306791Z outputs = self.model.decoder( 2025-11-03T16:30:54.5307144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5307496Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5307823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5308162Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5308530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5308888Z return func(*args, **kwargs) 2025-11-03T16:30:54.5309242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5309647Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5310033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5310389Z return func(*args, **kwargs) 2025-11-03T16:30:54.5310738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5311214Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5311419Z 2025-11-03T16:30:54.5311520Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5311869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5312180Z res = mod(**inputs) 2025-11-03T16:30:54.5312553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5312926Z outputs = self.model.decoder( 2025-11-03T16:30:54.5313471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5313865Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5314329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5314726Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5315126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5315494Z return func(*args, **kwargs) 2025-11-03T16:30:54.5315846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5316235Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5316620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5316980Z return func(*args, **kwargs) 2025-11-03T16:30:54.5317331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5317704Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5317835Z 2025-11-03T16:30:54.5317935Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5318282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5318590Z res = mod(**inputs) 2025-11-03T16:30:54.5319011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5319388Z outputs = self.model.decoder( 2025-11-03T16:30:54.5319748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5320115Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5320449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5320800Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5321159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5321517Z return func(*args, **kwargs) 2025-11-03T16:30:54.5321874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5322265Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5322648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5322996Z return func(*args, **kwargs) 2025-11-03T16:30:54.5323349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5323873Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5324008Z 2025-11-03T16:30:54.5324096Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5324328Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5324662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5324970Z res = mod(**inputs) 2025-11-03T16:30:54.5325309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5325681Z outputs = self.model.decoder( 2025-11-03T16:30:54.5326036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5326443Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5326767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5327104Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5327456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5327797Z return func(*args, **kwargs) 2025-11-03T16:30:54.5328137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5328514Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5328888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5329238Z return func(*args, **kwargs) 2025-11-03T16:30:54.5329575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5329951Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5330363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5330813Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5330984Z 2025-11-03T16:30:54.5331085Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5331415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5331715Z res = mod(**inputs) 2025-11-03T16:30:54.5332085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5332445Z outputs = self.model.decoder( 2025-11-03T16:30:54.5332794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5333152Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5333476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5333815Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5334168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5334505Z return func(*args, **kwargs) 2025-11-03T16:30:54.5334853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5335241Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5335627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5335987Z return func(*args, **kwargs) 2025-11-03T16:30:54.5336332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5336697Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5336823Z 2025-11-03T16:30:54.5336928Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5337260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5337552Z res = mod(**inputs) 2025-11-03T16:30:54.5337885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5338241Z outputs = self.model.decoder( 2025-11-03T16:30:54.5338593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5338947Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5339301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5339642Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5339996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5340350Z return func(*args, **kwargs) 2025-11-03T16:30:54.5340684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5341083Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5341252Z 2025-11-03T16:30:54.5341350Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5341686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5341989Z res = mod(**inputs) 2025-11-03T16:30:54.5342315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5342675Z outputs = self.model.decoder( 2025-11-03T16:30:54.5343025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5343385Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5343711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5344043Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5344394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5344741Z return func(*args, **kwargs) 2025-11-03T16:30:54.5345116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5345512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5345881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5346199Z return self.act(input) 2025-11-03T16:30:54.5346301Z 2025-11-03T16:30:54.5346406Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5346736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5347027Z res = mod(**inputs) 2025-11-03T16:30:54.5347357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5347719Z outputs = self.model.decoder( 2025-11-03T16:30:54.5348075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5348442Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5348769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5349124Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5349493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5349857Z return func(*args, **kwargs) 2025-11-03T16:30:54.5350205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5350584Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5350725Z 2025-11-03T16:30:54.5350825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5351171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5351484Z res = mod(**inputs) 2025-11-03T16:30:54.5351816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5352219Z outputs = self.model.decoder( 2025-11-03T16:30:54.5352580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5352950Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5353275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5353625Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5354080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5354464Z return func(*args, **kwargs) 2025-11-03T16:30:54.5354851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5355251Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5355655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5356021Z return func(*args, **kwargs) 2025-11-03T16:30:54.5356372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5356816Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5357013Z 2025-11-03T16:30:54.5357114Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5357455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5357762Z res = mod(**inputs) 2025-11-03T16:30:54.5358139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5358503Z outputs = self.model.decoder( 2025-11-03T16:30:54.5358867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5359244Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5359584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5359941Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5360302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5360666Z return func(*args, **kwargs) 2025-11-03T16:30:54.5361024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5361421Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5361809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5362164Z return func(*args, **kwargs) 2025-11-03T16:30:54.5362528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5362907Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5363037Z 2025-11-03T16:30:54.5363144Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5363485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5363797Z res = mod(**inputs) 2025-11-03T16:30:54.5364140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5364518Z outputs = self.model.decoder( 2025-11-03T16:30:54.5364888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5365260Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5365602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5365979Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5366339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5366694Z return func(*args, **kwargs) 2025-11-03T16:30:54.5367038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5367425Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5367806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5368164Z return func(*args, **kwargs) 2025-11-03T16:30:54.5368511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5368891Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5369035Z 2025-11-03T16:30:54.5369114Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5369347Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5369693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5369994Z res = mod(**inputs) 2025-11-03T16:30:54.5370335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5370703Z outputs = self.model.decoder( 2025-11-03T16:30:54.5371064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5371420Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5371801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5372152Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5372520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5372877Z return func(*args, **kwargs) 2025-11-03T16:30:54.5373224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5373619Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5374002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5374361Z return func(*args, **kwargs) 2025-11-03T16:30:54.5374727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5375115Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5375547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5376027Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5376204Z 2025-11-03T16:30:54.5376310Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5376650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5376954Z res = mod(**inputs) 2025-11-03T16:30:54.5377299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5377670Z outputs = self.model.decoder( 2025-11-03T16:30:54.5378032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5378395Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5378727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5379102Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5379460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5379812Z return func(*args, **kwargs) 2025-11-03T16:30:54.5380154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5380539Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5380914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5381263Z return func(*args, **kwargs) 2025-11-03T16:30:54.5381605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5381974Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5382115Z 2025-11-03T16:30:54.5382214Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5382561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5382908Z res = mod(**inputs) 2025-11-03T16:30:54.5383249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5383626Z outputs = self.model.decoder( 2025-11-03T16:30:54.5383998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5384365Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5384696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5385070Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5385435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5385794Z return func(*args, **kwargs) 2025-11-03T16:30:54.5386142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5386549Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5386720Z 2025-11-03T16:30:54.5386821Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5387173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5387480Z res = mod(**inputs) 2025-11-03T16:30:54.5387826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5388179Z outputs = self.model.decoder( 2025-11-03T16:30:54.5388528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5388889Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5389212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5389552Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5389906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5390260Z return func(*args, **kwargs) 2025-11-03T16:30:54.5390609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5391020Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5391388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5391717Z return self.act(input) 2025-11-03T16:30:54.5391828Z 2025-11-03T16:30:54.5391925Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5392304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5392611Z res = mod(**inputs) 2025-11-03T16:30:54.5392946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5393317Z outputs = self.model.decoder( 2025-11-03T16:30:54.5393678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5394137Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5394472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5394840Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5395220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5395590Z return func(*args, **kwargs) 2025-11-03T16:30:54.5395954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5396320Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5396457Z 2025-11-03T16:30:54.5396556Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5396893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5397197Z res = mod(**inputs) 2025-11-03T16:30:54.5397531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5397884Z outputs = self.model.decoder( 2025-11-03T16:30:54.5398277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5398640Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5398969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5399301Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5399659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5400006Z return func(*args, **kwargs) 2025-11-03T16:30:54.5400357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5400751Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5401117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5401470Z return func(*args, **kwargs) 2025-11-03T16:30:54.5401818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5402254Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5402443Z 2025-11-03T16:30:54.5402545Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5402872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5403173Z res = mod(**inputs) 2025-11-03T16:30:54.5403505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5403868Z outputs = self.model.decoder( 2025-11-03T16:30:54.5404213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5404574Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5404905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5405281Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5405636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5405975Z return func(*args, **kwargs) 2025-11-03T16:30:54.5406318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5406692Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5407061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5407403Z return func(*args, **kwargs) 2025-11-03T16:30:54.5407745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5408109Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5408242Z 2025-11-03T16:30:54.5408338Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5408673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5408967Z res = mod(**inputs) 2025-11-03T16:30:54.5409298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5409654Z outputs = self.model.decoder( 2025-11-03T16:30:54.5410005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5410363Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5410680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5411049Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5411402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5411747Z return func(*args, **kwargs) 2025-11-03T16:30:54.5412087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5412455Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5412822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5413166Z return func(*args, **kwargs) 2025-11-03T16:30:54.5413667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5414097Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5414243Z 2025-11-03T16:30:54.5414323Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5414561Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5414909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5415235Z res = mod(**inputs) 2025-11-03T16:30:54.5415562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5415923Z outputs = self.model.decoder( 2025-11-03T16:30:54.5416282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5416642Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5416962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5417302Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5417653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5418003Z return func(*args, **kwargs) 2025-11-03T16:30:54.5418354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5418800Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5419184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5419531Z return func(*args, **kwargs) 2025-11-03T16:30:54.5419879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5420259Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5420667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5421135Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5421321Z 2025-11-03T16:30:54.5421421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5421770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5422079Z res = mod(**inputs) 2025-11-03T16:30:54.5422418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5422786Z outputs = self.model.decoder( 2025-11-03T16:30:54.5423145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5423517Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5423838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5424188Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5424599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5424954Z return func(*args, **kwargs) 2025-11-03T16:30:54.5425306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5425688Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5426069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5426421Z return func(*args, **kwargs) 2025-11-03T16:30:54.5426770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5427136Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5427264Z 2025-11-03T16:30:54.5427362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5427701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5428007Z res = mod(**inputs) 2025-11-03T16:30:54.5428347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5428709Z outputs = self.model.decoder( 2025-11-03T16:30:54.5429067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5429430Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5429758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5430101Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5430451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5430805Z return func(*args, **kwargs) 2025-11-03T16:30:54.5431155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5431565Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5431764Z 2025-11-03T16:30:54.5431876Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5432228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5432547Z res = mod(**inputs) 2025-11-03T16:30:54.5432898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5433275Z outputs = self.model.decoder( 2025-11-03T16:30:54.5433637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5434087Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5434444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5434813Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5435044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5435124Z return func(*args, **kwargs) 2025-11-03T16:30:54.5435360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5435478Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5435694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5435764Z return self.act(input) 2025-11-03T16:30:54.5435768Z 2025-11-03T16:30:54.5435879Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5436103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5436166Z res = mod(**inputs) 2025-11-03T16:30:54.5436414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5436489Z outputs = self.model.decoder( 2025-11-03T16:30:54.5436734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5436802Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5437020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5437096Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5437328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5437403Z return func(*args, **kwargs) 2025-11-03T16:30:54.5437638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5437724Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5437730Z 2025-11-03T16:30:54.5437827Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5438014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5438080Z res = mod(**inputs) 2025-11-03T16:30:54.5438319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5438395Z outputs = self.model.decoder( 2025-11-03T16:30:54.5438631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5438709Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5438918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5438994Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5439235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5439333Z return func(*args, **kwargs) 2025-11-03T16:30:54.5439572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5439666Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5439895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5439970Z return func(*args, **kwargs) 2025-11-03T16:30:54.5440204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5440353Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5440359Z 2025-11-03T16:30:54.5440457Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5440651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5440713Z res = mod(**inputs) 2025-11-03T16:30:54.5440947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5441023Z outputs = self.model.decoder( 2025-11-03T16:30:54.5441258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5441330Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5441538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5441611Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5441884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5441951Z return func(*args, **kwargs) 2025-11-03T16:30:54.5442195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5442292Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5442517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5442588Z return func(*args, **kwargs) 2025-11-03T16:30:54.5442825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5442907Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5442910Z 2025-11-03T16:30:54.5443007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5443213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5443272Z res = mod(**inputs) 2025-11-03T16:30:54.5443502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5443580Z outputs = self.model.decoder( 2025-11-03T16:30:54.5443812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5443883Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5444087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5444160Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5444389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5444450Z return func(*args, **kwargs) 2025-11-03T16:30:54.5444691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5444781Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5445034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5445106Z return func(*args, **kwargs) 2025-11-03T16:30:54.5445332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5445421Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5445424Z 2025-11-03T16:30:54.5445500Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5445601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5445786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5445846Z res = mod(**inputs) 2025-11-03T16:30:54.5446085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5446152Z outputs = self.model.decoder( 2025-11-03T16:30:54.5446389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5446455Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5446659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5446741Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5446963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5447032Z return func(*args, **kwargs) 2025-11-03T16:30:54.5447262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5447383Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5447618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5447685Z return func(*args, **kwargs) 2025-11-03T16:30:54.5447922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5448014Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5448305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5448430Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5448434Z 2025-11-03T16:30:54.5448529Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5448722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5448784Z res = mod(**inputs) 2025-11-03T16:30:54.5449026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5449096Z outputs = self.model.decoder( 2025-11-03T16:30:54.5449329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5449404Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5449613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5449691Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5449923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5449993Z return func(*args, **kwargs) 2025-11-03T16:30:54.5450228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5450317Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5450548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5450653Z return func(*args, **kwargs) 2025-11-03T16:30:54.5450888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5450965Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5450968Z 2025-11-03T16:30:54.5451063Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5451253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5451312Z res = mod(**inputs) 2025-11-03T16:30:54.5451552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5451623Z outputs = self.model.decoder( 2025-11-03T16:30:54.5451852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5451927Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5452134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5452215Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5452441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5452509Z return func(*args, **kwargs) 2025-11-03T16:30:54.5452738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5452849Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5452852Z 2025-11-03T16:30:54.5452986Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5453169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5453239Z res = mod(**inputs) 2025-11-03T16:30:54.5453472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5453543Z outputs = self.model.decoder( 2025-11-03T16:30:54.5453782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5453851Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5454065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5454139Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5454372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5454441Z return func(*args, **kwargs) 2025-11-03T16:30:54.5454671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5454792Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5454994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5455069Z return self.act(input) 2025-11-03T16:30:54.5455072Z 2025-11-03T16:30:54.5455172Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5455357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5455429Z res = mod(**inputs) 2025-11-03T16:30:54.5455663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5455741Z outputs = self.model.decoder( 2025-11-03T16:30:54.5455978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5456048Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5456289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5456361Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5456594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5456659Z return func(*args, **kwargs) 2025-11-03T16:30:54.5456904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5456980Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5456984Z 2025-11-03T16:30:54.5457078Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5457274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5457333Z res = mod(**inputs) 2025-11-03T16:30:54.5457576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5457645Z outputs = self.model.decoder( 2025-11-03T16:30:54.5457884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5457958Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5458166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5458245Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5458472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5458535Z return func(*args, **kwargs) 2025-11-03T16:30:54.5458815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5458909Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5459142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5459206Z return func(*args, **kwargs) 2025-11-03T16:30:54.5459440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5459579Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5459583Z 2025-11-03T16:30:54.5459675Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5459866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5459926Z res = mod(**inputs) 2025-11-03T16:30:54.5460174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5460243Z outputs = self.model.decoder( 2025-11-03T16:30:54.5460483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5460570Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5460770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5460851Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5461073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5461141Z return func(*args, **kwargs) 2025-11-03T16:30:54.5461371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5461464Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5461694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5461788Z return func(*args, **kwargs) 2025-11-03T16:30:54.5462027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5462102Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5462105Z 2025-11-03T16:30:54.5462200Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5462391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5462450Z res = mod(**inputs) 2025-11-03T16:30:54.5462688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5462755Z outputs = self.model.decoder( 2025-11-03T16:30:54.5462995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5463061Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5463272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5463352Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5463583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5463653Z return func(*args, **kwargs) 2025-11-03T16:30:54.5463887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5463980Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5464218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5464324Z return func(*args, **kwargs) 2025-11-03T16:30:54.5464566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5464652Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5464655Z 2025-11-03T16:30:54.5464733Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5464838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5465026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5465095Z res = mod(**inputs) 2025-11-03T16:30:54.5465334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5465409Z outputs = self.model.decoder( 2025-11-03T16:30:54.5465646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5465717Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5465936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5466016Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5466253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5466317Z return func(*args, **kwargs) 2025-11-03T16:30:54.5466552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5466653Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5466881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5466953Z return func(*args, **kwargs) 2025-11-03T16:30:54.5467191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5467285Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5467629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5467758Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5467761Z 2025-11-03T16:30:54.5467865Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5468053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5468120Z res = mod(**inputs) 2025-11-03T16:30:54.5468360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5468430Z outputs = self.model.decoder( 2025-11-03T16:30:54.5468679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5468747Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5468973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5469046Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5469277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5469351Z return func(*args, **kwargs) 2025-11-03T16:30:54.5469584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5469683Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5469914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5469986Z return func(*args, **kwargs) 2025-11-03T16:30:54.5470261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5470343Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5470347Z 2025-11-03T16:30:54.5470451Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5470638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5470707Z res = mod(**inputs) 2025-11-03T16:30:54.5470948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5471016Z outputs = self.model.decoder( 2025-11-03T16:30:54.5471263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5471330Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5471553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5471627Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5471856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5471929Z return func(*args, **kwargs) 2025-11-03T16:30:54.5472163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5472283Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5472286Z 2025-11-03T16:30:54.5472385Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5472588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5472649Z res = mod(**inputs) 2025-11-03T16:30:54.5472891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5472968Z outputs = self.model.decoder( 2025-11-03T16:30:54.5473208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5473315Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5473523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5473597Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5473840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5473907Z return func(*args, **kwargs) 2025-11-03T16:30:54.5474230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5474348Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5474568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5474636Z return self.act(input) 2025-11-03T16:30:54.5474642Z 2025-11-03T16:30:54.5474743Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5474950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5475012Z res = mod(**inputs) 2025-11-03T16:30:54.5475267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5475340Z outputs = self.model.decoder( 2025-11-03T16:30:54.5475583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5475662Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5475911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5476014Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5476252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5476321Z return func(*args, **kwargs) 2025-11-03T16:30:54.5476570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5476649Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5476653Z 2025-11-03T16:30:54.5476759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5476955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5477023Z res = mod(**inputs) 2025-11-03T16:30:54.5477267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5477340Z outputs = self.model.decoder( 2025-11-03T16:30:54.5477589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5477662Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5477886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5477963Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5478201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5478276Z return func(*args, **kwargs) 2025-11-03T16:30:54.5478517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5478618Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5478857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5478930Z return func(*args, **kwargs) 2025-11-03T16:30:54.5479171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5479348Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5479352Z 2025-11-03T16:30:54.5479458Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5479651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5479720Z res = mod(**inputs) 2025-11-03T16:30:54.5479966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5480037Z outputs = self.model.decoder( 2025-11-03T16:30:54.5480291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5480360Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5480581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5480659Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5480896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5480971Z return func(*args, **kwargs) 2025-11-03T16:30:54.5481218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5481320Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5481555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5481628Z return func(*args, **kwargs) 2025-11-03T16:30:54.5481901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5481980Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5481986Z 2025-11-03T16:30:54.5482095Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5482283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5482351Z res = mod(**inputs) 2025-11-03T16:30:54.5482591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5482661Z outputs = self.model.decoder( 2025-11-03T16:30:54.5482910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5482980Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5483208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5483285Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5483524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5483592Z return func(*args, **kwargs) 2025-11-03T16:30:54.5483831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5483934Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5484168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5484242Z return func(*args, **kwargs) 2025-11-03T16:30:54.5484478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5484562Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5484568Z 2025-11-03T16:30:54.5484654Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5484753Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5484984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5485046Z res = mod(**inputs) 2025-11-03T16:30:54.5485290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5485367Z outputs = self.model.decoder( 2025-11-03T16:30:54.5485609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5485686Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5485902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5485983Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5486221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5486290Z return func(*args, **kwargs) 2025-11-03T16:30:54.5486540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5486635Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5486875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5486942Z return func(*args, **kwargs) 2025-11-03T16:30:54.5487182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5487287Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5487608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5487747Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5487751Z 2025-11-03T16:30:54.5487852Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5488051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5488113Z res = mod(**inputs) 2025-11-03T16:30:54.5488358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5488435Z outputs = self.model.decoder( 2025-11-03T16:30:54.5488676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5488751Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5488966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5489044Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5489296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5489367Z return func(*args, **kwargs) 2025-11-03T16:30:54.5489623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5489719Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5489953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5490026Z return func(*args, **kwargs) 2025-11-03T16:30:54.5490266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5490352Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5490355Z 2025-11-03T16:30:54.5490456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5490652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5490750Z res = mod(**inputs) 2025-11-03T16:30:54.5490996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5491073Z outputs = self.model.decoder( 2025-11-03T16:30:54.5491315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5491393Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5491609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5491684Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5491929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5491998Z return func(*args, **kwargs) 2025-11-03T16:30:54.5492246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5492375Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5492378Z 2025-11-03T16:30:54.5492481Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5492670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5492730Z res = mod(**inputs) 2025-11-03T16:30:54.5492976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5493046Z outputs = self.model.decoder( 2025-11-03T16:30:54.5493292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5493472Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5493898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5494012Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5494269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5494393Z return func(*args, **kwargs) 2025-11-03T16:30:54.5494650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5494800Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5495064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5495166Z return self.act(input) 2025-11-03T16:30:54.5495170Z 2025-11-03T16:30:54.5495321Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5495541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5495657Z res = mod(**inputs) 2025-11-03T16:30:54.5495906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5496057Z outputs = self.model.decoder( 2025-11-03T16:30:54.5496317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5496414Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5496703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5496800Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5497083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5497185Z return func(*args, **kwargs) 2025-11-03T16:30:54.5497443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5497604Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5497608Z 2025-11-03T16:30:54.5497728Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5497957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5498059Z res = mod(**inputs) 2025-11-03T16:30:54.5498361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5498452Z outputs = self.model.decoder( 2025-11-03T16:30:54.5498708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5498826Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5499052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5499207Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5499462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5499552Z return func(*args, **kwargs) 2025-11-03T16:30:54.5499824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5513715Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5514120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5514213Z return func(*args, **kwargs) 2025-11-03T16:30:54.5514481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 218, in forward 2025-11-03T16:30:54.5514812Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:30:54.5514828Z 2025-11-03T16:30:54.5514948Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5515179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5515257Z res = mod(**inputs) 2025-11-03T16:30:54.5515518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5515618Z outputs = self.model.decoder( 2025-11-03T16:30:54.5515864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5515938Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5516164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5516249Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5516494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5516567Z return func(*args, **kwargs) 2025-11-03T16:30:54.5516805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5516914Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5517148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5517223Z return func(*args, **kwargs) 2025-11-03T16:30:54.5517455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 237, in forward 2025-11-03T16:30:54.5517544Z key_states = self.k_proj(current_states) 2025-11-03T16:30:54.5517548Z 2025-11-03T16:30:54.5517649Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5517846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5517919Z res = mod(**inputs) 2025-11-03T16:30:54.5518227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5518309Z outputs = self.model.decoder( 2025-11-03T16:30:54.5518548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5518618Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5518841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5518916Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5519157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5519229Z return func(*args, **kwargs) 2025-11-03T16:30:54.5519459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5519566Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5519799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5519874Z return func(*args, **kwargs) 2025-11-03T16:30:54.5520109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 238, in forward 2025-11-03T16:30:54.5520203Z value_states = self.v_proj(current_states) 2025-11-03T16:30:54.5520207Z 2025-11-03T16:30:54.5520287Z cudagraph partition due to non gpu ops 2025-11-03T16:30:54.5520387Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5520587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5520697Z res = mod(**inputs) 2025-11-03T16:30:54.5520949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5521024Z outputs = self.model.decoder( 2025-11-03T16:30:54.5521262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5521339Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5521553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5521634Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5521863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5521928Z return func(*args, **kwargs) 2025-11-03T16:30:54.5522172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5522267Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5522506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5522575Z return func(*args, **kwargs) 2025-11-03T16:30:54.5522818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 256, in forward 2025-11-03T16:30:54.5522914Z attn_output, attn_weights = attention_interface( 2025-11-03T16:30:54.5523193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:30:54.5523334Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:30:54.5523338Z 2025-11-03T16:30:54.5523437Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5523634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5523696Z res = mod(**inputs) 2025-11-03T16:30:54.5523939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5524050Z outputs = self.model.decoder( 2025-11-03T16:30:54.5524291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5524369Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5524591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5524674Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5524907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5524974Z return func(*args, **kwargs) 2025-11-03T16:30:54.5525222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 416, in forward 2025-11-03T16:30:54.5525317Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:30:54.5525558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5525623Z return func(*args, **kwargs) 2025-11-03T16:30:54.5525860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 270, in forward 2025-11-03T16:30:54.5525945Z attn_output = self.out_proj(attn_output) 2025-11-03T16:30:54.5525949Z 2025-11-03T16:30:54.5526048Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5526242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5526302Z res = mod(**inputs) 2025-11-03T16:30:54.5527163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5527246Z outputs = self.model.decoder( 2025-11-03T16:30:54.5527486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5527570Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5527782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5527865Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5528094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5528159Z return func(*args, **kwargs) 2025-11-03T16:30:54.5528403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5528522Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5528528Z 2025-11-03T16:30:54.5528635Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5528825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5528889Z res = mod(**inputs) 2025-11-03T16:30:54.5529137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5529207Z outputs = self.model.decoder( 2025-11-03T16:30:54.5529452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5529521Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5529743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5529818Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5530049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5530123Z return func(*args, **kwargs) 2025-11-03T16:30:54.5530390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 448, in forward 2025-11-03T16:30:54.5530515Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:30:54.5530720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:30:54.5530788Z return self.act(input) 2025-11-03T16:30:54.5530792Z 2025-11-03T16:30:54.5530899Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5531089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5531158Z res = mod(**inputs) 2025-11-03T16:30:54.5531396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-11-03T16:30:54.5531468Z outputs = self.model.decoder( 2025-11-03T16:30:54.5531710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-11-03T16:30:54.5531782Z layer_outputs = decoder_layer( 2025-11-03T16:30:54.5531998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:30:54.5532073Z return super().__call__(*args, **kwargs) 2025-11-03T16:30:54.5532310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:30:54.5532378Z return func(*args, **kwargs) 2025-11-03T16:30:54.5532611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-11-03T16:30:54.5532697Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:30:54.5532701Z 2025-11-03T16:30:54.5532828Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5533026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5533090Z res = mod(**inputs) 2025-11-03T16:30:54.5533333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1918, in forward 2025-11-03T16:30:54.5533416Z logits = self.lm_head(outputs[0]) 2025-11-03T16:30:54.5533419Z 2025-11-03T16:30:54.5533518Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:30:54.5533712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:30:54.5533772Z res = mod(**inputs) 2025-11-03T16:30:54.5534020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1924, in forward 2025-11-03T16:30:54.5534161Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:30:54.5534164Z 2025-11-03T16:31:04.2325934Z Compilation time (from dynamo_timed): 15.285150281 2025-11-03T16:31:04.2520136Z pass 2025-11-03T16:31:04.2521875Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:31:04.2522778Z TIMING: _recursive_pre_grad_passes:0.00722 _recursive_joint_graph_passes:0.62688 _recursive_post_grad_passes:0.06018 async_compile.wait:0.82139 code_gen:8.67975 inductor_compile:9.92827 backend_compile:12.90934 gc:0.00196 entire_frame_compile:15.28515 total_wall_time:15.28515 2025-11-03T16:31:04.2528607Z STATS: call_* op count: 336 | FakeTensorMode.__torch_dispatch__:7258 | FakeTensor.__torch_dispatch__:4394 | ProxyTorchDispatchMode.__torch_dispatch__:2009 2025-11-03T16:31:04.2532685Z Dynamo produced 1 graphs covering 336 ops with 0 graph breaks (0 unique) 2025-11-03T16:31:06.5204123Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:31:06.5204977Z import pynvml # type: ignore[import] 2025-11-03T16:31:09.6562910Z 2025-11-03T16:31:10.6547877Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:31:10.6552068Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:31:10.6559072Z cpu eval BertForMaskedLM 2025-11-03T16:31:11.1206331Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:31:11.3427029Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:31:11.6159765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:31:18.5696095Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5696505Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5696756Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5697047Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5697278Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5697540Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5697776Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5698030Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5698267Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5698511Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5698749Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5698966Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5699227Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5699635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5699962Z res = mod(**inputs) 2025-11-03T16:31:18.5700354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5701110Z outputs = self.bert( 2025-11-03T16:31:18.5701475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5701865Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5702241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5702613Z layer_outputs = layer_module( 2025-11-03T16:31:18.5702951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5703313Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5703691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5704076Z return func(*args, **kwargs) 2025-11-03T16:31:18.5704452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5704829Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5705210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5705573Z return func(*args, **kwargs) 2025-11-03T16:31:18.5705933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5706332Z self_outputs = self.self( 2025-11-03T16:31:18.5706694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5707047Z return func(*args, **kwargs) 2025-11-03T16:31:18.5707395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.5707922Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.5708195Z 2025-11-03T16:31:18.5708305Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5708767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5709098Z res = mod(**inputs) 2025-11-03T16:31:18.5709463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5709848Z outputs = self.bert( 2025-11-03T16:31:18.5710219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5710614Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5711011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5711399Z layer_outputs = layer_module( 2025-11-03T16:31:18.5711758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5712130Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5712516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5712882Z return func(*args, **kwargs) 2025-11-03T16:31:18.5713418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5713831Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5714336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5714738Z return func(*args, **kwargs) 2025-11-03T16:31:18.5715135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5715574Z self_outputs = self.self( 2025-11-03T16:31:18.5715927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5716283Z return func(*args, **kwargs) 2025-11-03T16:31:18.5716763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.5717117Z self.key(current_states) 2025-11-03T16:31:18.5717234Z 2025-11-03T16:31:18.5717336Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5717685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5717994Z res = mod(**inputs) 2025-11-03T16:31:18.5718328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5718691Z outputs = self.bert( 2025-11-03T16:31:18.5719037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5719403Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5719768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5720123Z layer_outputs = layer_module( 2025-11-03T16:31:18.5720457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5720806Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5721170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5721527Z return func(*args, **kwargs) 2025-11-03T16:31:18.5721868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5722327Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5722693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5723115Z return func(*args, **kwargs) 2025-11-03T16:31:18.5723462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5723829Z self_outputs = self.self( 2025-11-03T16:31:18.5724173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5724528Z return func(*args, **kwargs) 2025-11-03T16:31:18.5724880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.5725297Z self.value(current_states) 2025-11-03T16:31:18.5725418Z 2025-11-03T16:31:18.5725496Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5725729Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5726065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5726364Z res = mod(**inputs) 2025-11-03T16:31:18.5726701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5727050Z outputs = self.bert( 2025-11-03T16:31:18.5727382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5727733Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5728082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5728435Z layer_outputs = layer_module( 2025-11-03T16:31:18.5728759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5729132Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5729486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5729824Z return func(*args, **kwargs) 2025-11-03T16:31:18.5730162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5730527Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5730881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5731217Z return func(*args, **kwargs) 2025-11-03T16:31:18.5731560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5731909Z self_outputs = self.self( 2025-11-03T16:31:18.5732245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5732587Z return func(*args, **kwargs) 2025-11-03T16:31:18.5732919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.5733335Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.5733517Z 2025-11-03T16:31:18.5733620Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5733964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5734273Z res = mod(**inputs) 2025-11-03T16:31:18.5734614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5734980Z outputs = self.bert( 2025-11-03T16:31:18.5735325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5735692Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5736044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5736448Z layer_outputs = layer_module( 2025-11-03T16:31:18.5736784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5737134Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5737501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5737852Z return func(*args, **kwargs) 2025-11-03T16:31:18.5738211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5738595Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5738980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5739335Z return func(*args, **kwargs) 2025-11-03T16:31:18.5739707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.5740148Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.5740594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.5740985Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5741122Z 2025-11-03T16:31:18.5741225Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5741585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5741904Z res = mod(**inputs) 2025-11-03T16:31:18.5742311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5742688Z outputs = self.bert( 2025-11-03T16:31:18.5743032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5743411Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5743788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5744162Z layer_outputs = layer_module( 2025-11-03T16:31:18.5744498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5744879Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5745275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5745653Z return func(*args, **kwargs) 2025-11-03T16:31:18.5746041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5746423Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5746836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5747236Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5747642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5748103Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5748533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.5748935Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5749084Z 2025-11-03T16:31:18.5749194Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5749584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5749942Z res = mod(**inputs) 2025-11-03T16:31:18.5750347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5750757Z outputs = self.bert( 2025-11-03T16:31:18.5751148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5751557Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5751970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5752377Z layer_outputs = layer_module( 2025-11-03T16:31:18.5752751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5753145Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5753564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5754054Z return func(*args, **kwargs) 2025-11-03T16:31:18.5754472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5754897Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5755336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5755773Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5756206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5756716Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5757230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.5757710Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.5758142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.5758508Z return self.act(input) 2025-11-03T16:31:18.5758637Z 2025-11-03T16:31:18.5758751Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5759139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5759487Z res = mod(**inputs) 2025-11-03T16:31:18.5759879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5760273Z outputs = self.bert( 2025-11-03T16:31:18.5760658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5761067Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5761471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5761874Z layer_outputs = layer_module( 2025-11-03T16:31:18.5762250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5762643Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5763044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5763452Z return func(*args, **kwargs) 2025-11-03T16:31:18.5763862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5764277Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5764712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5765143Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5765555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.5766070Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.5766503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.5766909Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5767057Z 2025-11-03T16:31:18.5767162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5767530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5767853Z res = mod(**inputs) 2025-11-03T16:31:18.5768218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5768600Z outputs = self.bert( 2025-11-03T16:31:18.5768965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5769355Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5769732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5770118Z layer_outputs = layer_module( 2025-11-03T16:31:18.5770469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5770837Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5771222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5771596Z return func(*args, **kwargs) 2025-11-03T16:31:18.5772006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5772402Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5772790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5773161Z return func(*args, **kwargs) 2025-11-03T16:31:18.5773531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5773914Z self_outputs = self.self( 2025-11-03T16:31:18.5774279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5774651Z return func(*args, **kwargs) 2025-11-03T16:31:18.5775013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.5775555Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.5775843Z 2025-11-03T16:31:18.5775956Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5776361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5776703Z res = mod(**inputs) 2025-11-03T16:31:18.5777077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5777479Z outputs = self.bert( 2025-11-03T16:31:18.5777858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5778275Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5778670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5779080Z layer_outputs = layer_module( 2025-11-03T16:31:18.5779454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5779926Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5780335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5780726Z return func(*args, **kwargs) 2025-11-03T16:31:18.5781121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5781540Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5781952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5782355Z return func(*args, **kwargs) 2025-11-03T16:31:18.5782740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5783150Z self_outputs = self.self( 2025-11-03T16:31:18.5783540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5783946Z return func(*args, **kwargs) 2025-11-03T16:31:18.5784331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.5784740Z self.key(current_states) 2025-11-03T16:31:18.5784871Z 2025-11-03T16:31:18.5784983Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5785368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5785707Z res = mod(**inputs) 2025-11-03T16:31:18.5786087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5786489Z outputs = self.bert( 2025-11-03T16:31:18.5786905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5787315Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5787713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5788120Z layer_outputs = layer_module( 2025-11-03T16:31:18.5788491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5788883Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5789288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5789676Z return func(*args, **kwargs) 2025-11-03T16:31:18.5790070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5790489Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5790898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5791286Z return func(*args, **kwargs) 2025-11-03T16:31:18.5791641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5792012Z self_outputs = self.self( 2025-11-03T16:31:18.5792369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5792730Z return func(*args, **kwargs) 2025-11-03T16:31:18.5793084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.5793456Z self.value(current_states) 2025-11-03T16:31:18.5793578Z 2025-11-03T16:31:18.5793659Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5793901Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5794367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5794765Z res = mod(**inputs) 2025-11-03T16:31:18.5795148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5795555Z outputs = self.bert( 2025-11-03T16:31:18.5795906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5796264Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5796628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5796992Z layer_outputs = layer_module( 2025-11-03T16:31:18.5797330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5797681Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5798035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5798396Z return func(*args, **kwargs) 2025-11-03T16:31:18.5798747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5799126Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5799483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5799839Z return func(*args, **kwargs) 2025-11-03T16:31:18.5800187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5800548Z self_outputs = self.self( 2025-11-03T16:31:18.5800919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5801265Z return func(*args, **kwargs) 2025-11-03T16:31:18.5801623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.5802046Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.5802220Z 2025-11-03T16:31:18.5802331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5802674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5802977Z res = mod(**inputs) 2025-11-03T16:31:18.5803318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5803684Z outputs = self.bert( 2025-11-03T16:31:18.5804015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5804362Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5804713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5805068Z layer_outputs = layer_module( 2025-11-03T16:31:18.5805394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5805733Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5806079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5806426Z return func(*args, **kwargs) 2025-11-03T16:31:18.5806767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5807131Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5807489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5807824Z return func(*args, **kwargs) 2025-11-03T16:31:18.5808197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.5808605Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.5809010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.5809369Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5809506Z 2025-11-03T16:31:18.5809602Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5809939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5810242Z res = mod(**inputs) 2025-11-03T16:31:18.5810577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5810922Z outputs = self.bert( 2025-11-03T16:31:18.5811256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5811614Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5811966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5812321Z layer_outputs = layer_module( 2025-11-03T16:31:18.5812641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5812980Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5813516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5813874Z return func(*args, **kwargs) 2025-11-03T16:31:18.5814285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5814661Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5815043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5815414Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5815798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5816221Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5816624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.5816992Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5817122Z 2025-11-03T16:31:18.5817233Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5817574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5817868Z res = mod(**inputs) 2025-11-03T16:31:18.5818206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5818559Z outputs = self.bert( 2025-11-03T16:31:18.5818895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5819249Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5819610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5819963Z layer_outputs = layer_module( 2025-11-03T16:31:18.5820288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5820629Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5820975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5821366Z return func(*args, **kwargs) 2025-11-03T16:31:18.5821710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5822076Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5822449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5822817Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5823200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5823625Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5824025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.5824408Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.5824767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.5825085Z return self.act(input) 2025-11-03T16:31:18.5825190Z 2025-11-03T16:31:18.5825295Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5825632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5825925Z res = mod(**inputs) 2025-11-03T16:31:18.5826258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5826608Z outputs = self.bert( 2025-11-03T16:31:18.5826938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5827319Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5827664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5828018Z layer_outputs = layer_module( 2025-11-03T16:31:18.5828346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5828688Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5829034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5829386Z return func(*args, **kwargs) 2025-11-03T16:31:18.5829730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5830102Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5830483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5830857Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5831243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.5831677Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.5832096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.5832472Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5832601Z 2025-11-03T16:31:18.5832701Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5833046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5833355Z res = mod(**inputs) 2025-11-03T16:31:18.5833702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5834122Z outputs = self.bert( 2025-11-03T16:31:18.5834488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5834918Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5835298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5835693Z layer_outputs = layer_module( 2025-11-03T16:31:18.5836049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5836401Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5836767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5837129Z return func(*args, **kwargs) 2025-11-03T16:31:18.5837483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5837861Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5838233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5838590Z return func(*args, **kwargs) 2025-11-03T16:31:18.5838941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5839296Z self_outputs = self.self( 2025-11-03T16:31:18.5839643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5840002Z return func(*args, **kwargs) 2025-11-03T16:31:18.5840352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.5840882Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.5841133Z 2025-11-03T16:31:18.5841237Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5841586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5841895Z res = mod(**inputs) 2025-11-03T16:31:18.5842237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5842600Z outputs = self.bert( 2025-11-03T16:31:18.5842937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5843306Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5843667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5844034Z layer_outputs = layer_module( 2025-11-03T16:31:18.5844362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5844720Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5845084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5845445Z return func(*args, **kwargs) 2025-11-03T16:31:18.5845797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5846164Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5846533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5846894Z return func(*args, **kwargs) 2025-11-03T16:31:18.5847245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5847601Z self_outputs = self.self( 2025-11-03T16:31:18.5847958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5849098Z return func(*args, **kwargs) 2025-11-03T16:31:18.5849443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.5849794Z self.key(current_states) 2025-11-03T16:31:18.5849902Z 2025-11-03T16:31:18.5849998Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5850344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5850645Z res = mod(**inputs) 2025-11-03T16:31:18.5850980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5851333Z outputs = self.bert( 2025-11-03T16:31:18.5851661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5852023Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5852376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5852728Z layer_outputs = layer_module( 2025-11-03T16:31:18.5853045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5853384Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5853738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5854083Z return func(*args, **kwargs) 2025-11-03T16:31:18.5854423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5854814Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5855177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5855526Z return func(*args, **kwargs) 2025-11-03T16:31:18.5855871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5856218Z self_outputs = self.self( 2025-11-03T16:31:18.5856559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5856905Z return func(*args, **kwargs) 2025-11-03T16:31:18.5857250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.5857604Z self.value(current_states) 2025-11-03T16:31:18.5857715Z 2025-11-03T16:31:18.5857795Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5858025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5858364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5858673Z res = mod(**inputs) 2025-11-03T16:31:18.5859005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5859360Z outputs = self.bert( 2025-11-03T16:31:18.5859694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5860056Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5860409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5860757Z layer_outputs = layer_module( 2025-11-03T16:31:18.5861085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5861427Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5861784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5862161Z return func(*args, **kwargs) 2025-11-03T16:31:18.5862501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5862881Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5863243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5863601Z return func(*args, **kwargs) 2025-11-03T16:31:18.5863943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5864304Z self_outputs = self.self( 2025-11-03T16:31:18.5864654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5865006Z return func(*args, **kwargs) 2025-11-03T16:31:18.5865361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.5865770Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.5865953Z 2025-11-03T16:31:18.5866052Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5866396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5866701Z res = mod(**inputs) 2025-11-03T16:31:18.5867043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5867397Z outputs = self.bert( 2025-11-03T16:31:18.5867798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5868169Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5868533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5868894Z layer_outputs = layer_module( 2025-11-03T16:31:18.5869231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5869586Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5869950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5870309Z return func(*args, **kwargs) 2025-11-03T16:31:18.5870657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5871033Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5871404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5871762Z return func(*args, **kwargs) 2025-11-03T16:31:18.5872117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.5872548Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.5872969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.5873352Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5873487Z 2025-11-03T16:31:18.5873598Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5874018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5874347Z res = mod(**inputs) 2025-11-03T16:31:18.5874703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5875072Z outputs = self.bert( 2025-11-03T16:31:18.5875467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5875845Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5876219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5876595Z layer_outputs = layer_module( 2025-11-03T16:31:18.5876937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5877291Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5877665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5878031Z return func(*args, **kwargs) 2025-11-03T16:31:18.5878397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5878788Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5879179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5879567Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5879973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5880427Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5880847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.5881229Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5881375Z 2025-11-03T16:31:18.5881513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5881874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5882205Z res = mod(**inputs) 2025-11-03T16:31:18.5882555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5882950Z outputs = self.bert( 2025-11-03T16:31:18.5883309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5883748Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5884124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5884497Z layer_outputs = layer_module( 2025-11-03T16:31:18.5884846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5885216Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5885595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5885977Z return func(*args, **kwargs) 2025-11-03T16:31:18.5886340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5886733Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5887135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5887529Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5887932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5888385Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5888812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.5889237Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.5889649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.5889981Z return self.act(input) 2025-11-03T16:31:18.5890099Z 2025-11-03T16:31:18.5890200Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5890553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5890876Z res = mod(**inputs) 2025-11-03T16:31:18.5891226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5891589Z outputs = self.bert( 2025-11-03T16:31:18.5891984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5892340Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5892688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5893039Z layer_outputs = layer_module( 2025-11-03T16:31:18.5893369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5893706Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5894059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5894407Z return func(*args, **kwargs) 2025-11-03T16:31:18.5894743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5895110Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5895528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5895897Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5896282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.5896711Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.5897120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.5897494Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5897623Z 2025-11-03T16:31:18.5897727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5898062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5898355Z res = mod(**inputs) 2025-11-03T16:31:18.5898693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5899045Z outputs = self.bert( 2025-11-03T16:31:18.5899381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5899730Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5900086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5900441Z layer_outputs = layer_module( 2025-11-03T16:31:18.5900766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5901103Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5901451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5901801Z return func(*args, **kwargs) 2025-11-03T16:31:18.5902147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5902548Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5902895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5903244Z return func(*args, **kwargs) 2025-11-03T16:31:18.5903592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5903943Z self_outputs = self.self( 2025-11-03T16:31:18.5904282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5904619Z return func(*args, **kwargs) 2025-11-03T16:31:18.5904962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.5905452Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.5905696Z 2025-11-03T16:31:18.5905800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5906131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5906421Z res = mod(**inputs) 2025-11-03T16:31:18.5906752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5907104Z outputs = self.bert( 2025-11-03T16:31:18.5907435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5907783Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5908130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5908521Z layer_outputs = layer_module( 2025-11-03T16:31:18.5908845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5909194Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5909547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5909904Z return func(*args, **kwargs) 2025-11-03T16:31:18.5910261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5910629Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5910994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5911342Z return func(*args, **kwargs) 2025-11-03T16:31:18.5911696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5912058Z self_outputs = self.self( 2025-11-03T16:31:18.5912405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5912755Z return func(*args, **kwargs) 2025-11-03T16:31:18.5913107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.5913639Z self.key(current_states) 2025-11-03T16:31:18.5913754Z 2025-11-03T16:31:18.5913866Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5914279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5914607Z res = mod(**inputs) 2025-11-03T16:31:18.5914975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5915379Z outputs = self.bert( 2025-11-03T16:31:18.5915727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5916166Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5916524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5916887Z layer_outputs = layer_module( 2025-11-03T16:31:18.5917219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5917567Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5917922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5918281Z return func(*args, **kwargs) 2025-11-03T16:31:18.5918635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5919013Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5919377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5919727Z return func(*args, **kwargs) 2025-11-03T16:31:18.5920081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5920443Z self_outputs = self.self( 2025-11-03T16:31:18.5920787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5921135Z return func(*args, **kwargs) 2025-11-03T16:31:18.5921489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.5921856Z self.value(current_states) 2025-11-03T16:31:18.5921966Z 2025-11-03T16:31:18.5922104Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5922337Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5922674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5922985Z res = mod(**inputs) 2025-11-03T16:31:18.5923323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5923684Z outputs = self.bert( 2025-11-03T16:31:18.5924016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5924384Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5924745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5925112Z layer_outputs = layer_module( 2025-11-03T16:31:18.5925446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5925789Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5926157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5926511Z return func(*args, **kwargs) 2025-11-03T16:31:18.5926868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5927245Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5927603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5927958Z return func(*args, **kwargs) 2025-11-03T16:31:18.5928321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5928672Z self_outputs = self.self( 2025-11-03T16:31:18.5929004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5929384Z return func(*args, **kwargs) 2025-11-03T16:31:18.5929728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.5930144Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.5930319Z 2025-11-03T16:31:18.5930424Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5930756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5931057Z res = mod(**inputs) 2025-11-03T16:31:18.5931394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5931742Z outputs = self.bert( 2025-11-03T16:31:18.5932071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5932429Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5932783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5933138Z layer_outputs = layer_module( 2025-11-03T16:31:18.5933463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5933798Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5934154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5934504Z return func(*args, **kwargs) 2025-11-03T16:31:18.5934847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5935247Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5935596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5935943Z return func(*args, **kwargs) 2025-11-03T16:31:18.5936281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.5936690Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.5937084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.5937450Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5937586Z 2025-11-03T16:31:18.5937682Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5938016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5938318Z res = mod(**inputs) 2025-11-03T16:31:18.5938642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5938994Z outputs = self.bert( 2025-11-03T16:31:18.5939324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5939679Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5940029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5940381Z layer_outputs = layer_module( 2025-11-03T16:31:18.5940710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5941064Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5941422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5941792Z return func(*args, **kwargs) 2025-11-03T16:31:18.5942144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5942548Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5942928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5943546Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5944435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5944889Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5945298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.5945671Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5945841Z 2025-11-03T16:31:18.5946004Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5946390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5946694Z res = mod(**inputs) 2025-11-03T16:31:18.5947034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5947389Z outputs = self.bert( 2025-11-03T16:31:18.5947723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5948077Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5948427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5948783Z layer_outputs = layer_module( 2025-11-03T16:31:18.5949159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5949503Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5949857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5950204Z return func(*args, **kwargs) 2025-11-03T16:31:18.5950549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5950916Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5951294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5951659Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5952125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.5952807Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.5953215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.5953612Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.5954073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.5954432Z return self.act(input) 2025-11-03T16:31:18.5954554Z 2025-11-03T16:31:18.5954661Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5955031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5955349Z res = mod(**inputs) 2025-11-03T16:31:18.5955675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5956031Z outputs = self.bert( 2025-11-03T16:31:18.5956374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5956732Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5957095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5957496Z layer_outputs = layer_module( 2025-11-03T16:31:18.5957824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5958160Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5958519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5958871Z return func(*args, **kwargs) 2025-11-03T16:31:18.5959218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.5959590Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.5959965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.5960432Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.5960822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.5961274Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.5961680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.5962042Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.5962177Z 2025-11-03T16:31:18.5962275Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5962612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5962917Z res = mod(**inputs) 2025-11-03T16:31:18.5963289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5963655Z outputs = self.bert( 2025-11-03T16:31:18.5963986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5964355Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5964718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5965073Z layer_outputs = layer_module( 2025-11-03T16:31:18.5965409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5965755Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5966122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5966479Z return func(*args, **kwargs) 2025-11-03T16:31:18.5966832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5967209Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5967582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5967953Z return func(*args, **kwargs) 2025-11-03T16:31:18.5968308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5968671Z self_outputs = self.self( 2025-11-03T16:31:18.5969022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5969380Z return func(*args, **kwargs) 2025-11-03T16:31:18.5969735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.5970228Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.5970532Z 2025-11-03T16:31:18.5970635Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5970985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5971295Z res = mod(**inputs) 2025-11-03T16:31:18.5971632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5972003Z outputs = self.bert( 2025-11-03T16:31:18.5972350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5972718Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5973086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5973444Z layer_outputs = layer_module( 2025-11-03T16:31:18.5973782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5974132Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5974491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5974850Z return func(*args, **kwargs) 2025-11-03T16:31:18.5975193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5975569Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5975936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5976296Z return func(*args, **kwargs) 2025-11-03T16:31:18.5976672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5977035Z self_outputs = self.self( 2025-11-03T16:31:18.5977382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5977741Z return func(*args, **kwargs) 2025-11-03T16:31:18.5978090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.5978448Z self.key(current_states) 2025-11-03T16:31:18.5978567Z 2025-11-03T16:31:18.5978667Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5979010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5979317Z res = mod(**inputs) 2025-11-03T16:31:18.5979653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5980011Z outputs = self.bert( 2025-11-03T16:31:18.5980349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5980715Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5981069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5981425Z layer_outputs = layer_module( 2025-11-03T16:31:18.5981757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5982102Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5982468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5982846Z return func(*args, **kwargs) 2025-11-03T16:31:18.5983223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5983604Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5984022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5984395Z return func(*args, **kwargs) 2025-11-03T16:31:18.5984755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5985129Z self_outputs = self.self( 2025-11-03T16:31:18.5985487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5985850Z return func(*args, **kwargs) 2025-11-03T16:31:18.5986210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.5986577Z self.value(current_states) 2025-11-03T16:31:18.5986707Z 2025-11-03T16:31:18.5986789Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.5987027Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5987385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5987695Z res = mod(**inputs) 2025-11-03T16:31:18.5988047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5988422Z outputs = self.bert( 2025-11-03T16:31:18.5988778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5989162Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5989524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5989899Z layer_outputs = layer_module( 2025-11-03T16:31:18.5990279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5990640Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5991018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5991380Z return func(*args, **kwargs) 2025-11-03T16:31:18.5991741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.5992126Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.5992506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5992866Z return func(*args, **kwargs) 2025-11-03T16:31:18.5993211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.5993570Z self_outputs = self.self( 2025-11-03T16:31:18.5993909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5994330Z return func(*args, **kwargs) 2025-11-03T16:31:18.5994676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.5995106Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.5995292Z 2025-11-03T16:31:18.5995660Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.5995998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.5996298Z res = mod(**inputs) 2025-11-03T16:31:18.5996625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.5996983Z outputs = self.bert( 2025-11-03T16:31:18.5997328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.5997688Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.5998091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.5998444Z layer_outputs = layer_module( 2025-11-03T16:31:18.5998769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.5999110Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.5999463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.5999800Z return func(*args, **kwargs) 2025-11-03T16:31:18.6000143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6000509Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6000864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6001215Z return func(*args, **kwargs) 2025-11-03T16:31:18.6001551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6001958Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6002363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6002730Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6002860Z 2025-11-03T16:31:18.6002960Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6003292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6003634Z res = mod(**inputs) 2025-11-03T16:31:18.6003971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6004328Z outputs = self.bert( 2025-11-03T16:31:18.6004662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6005019Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6005372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6005732Z layer_outputs = layer_module( 2025-11-03T16:31:18.6006058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6006398Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6006757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6007108Z return func(*args, **kwargs) 2025-11-03T16:31:18.6007455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6007832Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6008216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6008589Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6008986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6009418Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6009814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6010183Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6010319Z 2025-11-03T16:31:18.6010417Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6010759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6011104Z res = mod(**inputs) 2025-11-03T16:31:18.6011431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6011780Z outputs = self.bert( 2025-11-03T16:31:18.6012115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6012479Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6012824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6013181Z layer_outputs = layer_module( 2025-11-03T16:31:18.6013677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6014026Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6014390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6014740Z return func(*args, **kwargs) 2025-11-03T16:31:18.6015086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6015455Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6015832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6016207Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6016584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6017096Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6017496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6017897Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6018246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6018571Z return self.act(input) 2025-11-03T16:31:18.6018684Z 2025-11-03T16:31:18.6018781Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6019123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6019426Z res = mod(**inputs) 2025-11-03T16:31:18.6019751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6020108Z outputs = self.bert( 2025-11-03T16:31:18.6020446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6020803Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6021151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6021510Z layer_outputs = layer_module( 2025-11-03T16:31:18.6021837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6022184Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6022540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6022883Z return func(*args, **kwargs) 2025-11-03T16:31:18.6023227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6023594Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6023968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6024388Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6024766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6025202Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6025609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6025973Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6026105Z 2025-11-03T16:31:18.6026211Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6026543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6026847Z res = mod(**inputs) 2025-11-03T16:31:18.6027185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6027538Z outputs = self.bert( 2025-11-03T16:31:18.6027864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6028224Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6028576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6028931Z layer_outputs = layer_module( 2025-11-03T16:31:18.6029258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6029595Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6029989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6030346Z return func(*args, **kwargs) 2025-11-03T16:31:18.6030692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6031053Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6031413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6031760Z return func(*args, **kwargs) 2025-11-03T16:31:18.6032107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6032460Z self_outputs = self.self( 2025-11-03T16:31:18.6032790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6033137Z return func(*args, **kwargs) 2025-11-03T16:31:18.6033483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6034050Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6034343Z 2025-11-03T16:31:18.6034454Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6034810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6035136Z res = mod(**inputs) 2025-11-03T16:31:18.6035483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6035838Z outputs = self.bert( 2025-11-03T16:31:18.6036171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6036539Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6036900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6037262Z layer_outputs = layer_module( 2025-11-03T16:31:18.6037589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6037964Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6038327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6038677Z return func(*args, **kwargs) 2025-11-03T16:31:18.6039021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6039387Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6039736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6040085Z return func(*args, **kwargs) 2025-11-03T16:31:18.6040440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6040799Z self_outputs = self.self( 2025-11-03T16:31:18.6041129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6041477Z return func(*args, **kwargs) 2025-11-03T16:31:18.6041819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6042172Z self.key(current_states) 2025-11-03T16:31:18.6042280Z 2025-11-03T16:31:18.6042385Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6042715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6043018Z res = mod(**inputs) 2025-11-03T16:31:18.6043385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6043839Z outputs = self.bert( 2025-11-03T16:31:18.6044170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6044537Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6044891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6045249Z layer_outputs = layer_module( 2025-11-03T16:31:18.6045576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6045910Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6046268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6046622Z return func(*args, **kwargs) 2025-11-03T16:31:18.6046968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6047343Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6047705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6048057Z return func(*args, **kwargs) 2025-11-03T16:31:18.6048403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6048758Z self_outputs = self.self( 2025-11-03T16:31:18.6049090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6049441Z return func(*args, **kwargs) 2025-11-03T16:31:18.6049789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6050150Z self.value(current_states) 2025-11-03T16:31:18.6050264Z 2025-11-03T16:31:18.6050350Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6050573Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6050960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6051263Z res = mod(**inputs) 2025-11-03T16:31:18.6051600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6051957Z outputs = self.bert( 2025-11-03T16:31:18.6052293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6052660Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6053017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6053376Z layer_outputs = layer_module( 2025-11-03T16:31:18.6053703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6054052Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6054410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6054762Z return func(*args, **kwargs) 2025-11-03T16:31:18.6055109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6055471Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6055836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6056182Z return func(*args, **kwargs) 2025-11-03T16:31:18.6056524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6056904Z self_outputs = self.self( 2025-11-03T16:31:18.6057246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6057593Z return func(*args, **kwargs) 2025-11-03T16:31:18.6057934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6058343Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6058513Z 2025-11-03T16:31:18.6058613Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6058949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6059253Z res = mod(**inputs) 2025-11-03T16:31:18.6059592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6059946Z outputs = self.bert( 2025-11-03T16:31:18.6060274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6060635Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6060988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6061344Z layer_outputs = layer_module( 2025-11-03T16:31:18.6061663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6062006Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6062364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6062711Z return func(*args, **kwargs) 2025-11-03T16:31:18.6063055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6063411Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6063768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6064145Z return func(*args, **kwargs) 2025-11-03T16:31:18.6064488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6064883Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6065284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6065650Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6065777Z 2025-11-03T16:31:18.6065882Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6066215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6066509Z res = mod(**inputs) 2025-11-03T16:31:18.6066840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6067196Z outputs = self.bert( 2025-11-03T16:31:18.6067527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6067882Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6068225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6068580Z layer_outputs = layer_module( 2025-11-03T16:31:18.6068904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6069242Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6069621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6069972Z return func(*args, **kwargs) 2025-11-03T16:31:18.6070315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6070686Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6071060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6071421Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6071805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6072234Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6072634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6073005Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6073133Z 2025-11-03T16:31:18.6073230Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6073573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6073875Z res = mod(**inputs) 2025-11-03T16:31:18.6074274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6074624Z outputs = self.bert( 2025-11-03T16:31:18.6074979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6075360Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6075736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6076164Z layer_outputs = layer_module( 2025-11-03T16:31:18.6076511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6076875Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6077291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6077660Z return func(*args, **kwargs) 2025-11-03T16:31:18.6078024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6078406Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6078802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6079191Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6079594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6080042Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6080463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6080884Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6081259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6081600Z return self.act(input) 2025-11-03T16:31:18.6081710Z 2025-11-03T16:31:18.6081813Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6082176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6082503Z res = mod(**inputs) 2025-11-03T16:31:18.6082857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6083235Z outputs = self.bert( 2025-11-03T16:31:18.6083658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6084048Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6084422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6084802Z layer_outputs = layer_module( 2025-11-03T16:31:18.6085145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6085510Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6085886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6086260Z return func(*args, **kwargs) 2025-11-03T16:31:18.6086628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6087025Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6087426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6087827Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6088235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6088696Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6089120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6089509Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6089652Z 2025-11-03T16:31:18.6089759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6090117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6090426Z res = mod(**inputs) 2025-11-03T16:31:18.6090760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6091155Z outputs = self.bert( 2025-11-03T16:31:18.6091492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6091851Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6092196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6092550Z layer_outputs = layer_module( 2025-11-03T16:31:18.6092875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6093215Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6093578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6093919Z return func(*args, **kwargs) 2025-11-03T16:31:18.6094266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6094631Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6094991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6095336Z return func(*args, **kwargs) 2025-11-03T16:31:18.6095685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6096039Z self_outputs = self.self( 2025-11-03T16:31:18.6096378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6096725Z return func(*args, **kwargs) 2025-11-03T16:31:18.6097100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6097587Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6097838Z 2025-11-03T16:31:18.6097935Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6098273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6098575Z res = mod(**inputs) 2025-11-03T16:31:18.6098904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6099260Z outputs = self.bert( 2025-11-03T16:31:18.6099597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6099957Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6100305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6100665Z layer_outputs = layer_module( 2025-11-03T16:31:18.6100995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6101338Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6101695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6102038Z return func(*args, **kwargs) 2025-11-03T16:31:18.6102390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6102760Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6103120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6103471Z return func(*args, **kwargs) 2025-11-03T16:31:18.6103815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6104209Z self_outputs = self.self( 2025-11-03T16:31:18.6104551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6104895Z return func(*args, **kwargs) 2025-11-03T16:31:18.6105234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6105591Z self.key(current_states) 2025-11-03T16:31:18.6105705Z 2025-11-03T16:31:18.6105802Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6106141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6106443Z res = mod(**inputs) 2025-11-03T16:31:18.6106769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6107121Z outputs = self.bert( 2025-11-03T16:31:18.6107455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6107811Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6108151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6108508Z layer_outputs = layer_module( 2025-11-03T16:31:18.6108836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6109173Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6109525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6109861Z return func(*args, **kwargs) 2025-11-03T16:31:18.6110231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6110601Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6110957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6111302Z return func(*args, **kwargs) 2025-11-03T16:31:18.6111636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6111984Z self_outputs = self.self( 2025-11-03T16:31:18.6112320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6112663Z return func(*args, **kwargs) 2025-11-03T16:31:18.6112996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6113528Z self.value(current_states) 2025-11-03T16:31:18.6113651Z 2025-11-03T16:31:18.6113732Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6114004Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6114374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6114676Z res = mod(**inputs) 2025-11-03T16:31:18.6115035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6115388Z outputs = self.bert( 2025-11-03T16:31:18.6115723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6116072Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6116426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6116790Z layer_outputs = layer_module( 2025-11-03T16:31:18.6117119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6117523Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6117869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6118213Z return func(*args, **kwargs) 2025-11-03T16:31:18.6118556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6118921Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6119272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6119621Z return func(*args, **kwargs) 2025-11-03T16:31:18.6119974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6120331Z self_outputs = self.self( 2025-11-03T16:31:18.6120675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6121016Z return func(*args, **kwargs) 2025-11-03T16:31:18.6121364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6121773Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6121945Z 2025-11-03T16:31:18.6122051Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6122390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6122683Z res = mod(**inputs) 2025-11-03T16:31:18.6123014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6123417Z outputs = self.bert( 2025-11-03T16:31:18.6123754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6124111Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6124462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6124826Z layer_outputs = layer_module( 2025-11-03T16:31:18.6125150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6125488Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6125833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6126177Z return func(*args, **kwargs) 2025-11-03T16:31:18.6126525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6126891Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6127251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6127601Z return func(*args, **kwargs) 2025-11-03T16:31:18.6127942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6128350Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6128752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6129111Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6129247Z 2025-11-03T16:31:18.6129343Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6129681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6129983Z res = mod(**inputs) 2025-11-03T16:31:18.6130316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6130704Z outputs = self.bert( 2025-11-03T16:31:18.6131040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6131406Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6131760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6132115Z layer_outputs = layer_module( 2025-11-03T16:31:18.6132434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6132774Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6133129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6133479Z return func(*args, **kwargs) 2025-11-03T16:31:18.6133816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6134190Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6134572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6134943Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6135330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6135755Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6136159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6136566Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6136696Z 2025-11-03T16:31:18.6136800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6137138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6137434Z res = mod(**inputs) 2025-11-03T16:31:18.6137770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6138123Z outputs = self.bert( 2025-11-03T16:31:18.6138457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6138804Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6139154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6139561Z layer_outputs = layer_module( 2025-11-03T16:31:18.6139900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6140258Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6140610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6140961Z return func(*args, **kwargs) 2025-11-03T16:31:18.6141307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6141676Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6142056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6142420Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6142804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6143234Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6143645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6144072Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6144435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6144764Z return self.act(input) 2025-11-03T16:31:18.6144869Z 2025-11-03T16:31:18.6144977Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6145320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6145631Z res = mod(**inputs) 2025-11-03T16:31:18.6145963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6146314Z outputs = self.bert( 2025-11-03T16:31:18.6146651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6147022Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6147375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6147736Z layer_outputs = layer_module( 2025-11-03T16:31:18.6148068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6148412Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6148772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6149128Z return func(*args, **kwargs) 2025-11-03T16:31:18.6149528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6149915Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6150305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6150680Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6151073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6151527Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6151947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6152324Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6152455Z 2025-11-03T16:31:18.6152555Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6152904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6153214Z res = mod(**inputs) 2025-11-03T16:31:18.6153556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6153913Z outputs = self.bert( 2025-11-03T16:31:18.6154339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6154719Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6155085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6155454Z layer_outputs = layer_module( 2025-11-03T16:31:18.6155786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6156139Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6156512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6156881Z return func(*args, **kwargs) 2025-11-03T16:31:18.6157276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6157658Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6158033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6158396Z return func(*args, **kwargs) 2025-11-03T16:31:18.6158755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6159120Z self_outputs = self.self( 2025-11-03T16:31:18.6159477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6159841Z return func(*args, **kwargs) 2025-11-03T16:31:18.6160202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6160705Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6160959Z 2025-11-03T16:31:18.6161063Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6161419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6161734Z res = mod(**inputs) 2025-11-03T16:31:18.6162082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6162447Z outputs = self.bert( 2025-11-03T16:31:18.6162786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6163155Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6163557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6163924Z layer_outputs = layer_module( 2025-11-03T16:31:18.6164255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6164602Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6164962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6165325Z return func(*args, **kwargs) 2025-11-03T16:31:18.6165676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6166045Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6166415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6166776Z return func(*args, **kwargs) 2025-11-03T16:31:18.6167130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6167496Z self_outputs = self.self( 2025-11-03T16:31:18.6167842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6168199Z return func(*args, **kwargs) 2025-11-03T16:31:18.6168552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6168913Z self.key(current_states) 2025-11-03T16:31:18.6169022Z 2025-11-03T16:31:18.6169122Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6169470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6169778Z res = mod(**inputs) 2025-11-03T16:31:18.6170121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6170478Z outputs = self.bert( 2025-11-03T16:31:18.6170852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6171223Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6171597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6171966Z layer_outputs = layer_module( 2025-11-03T16:31:18.6172299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6172655Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6173030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6173398Z return func(*args, **kwargs) 2025-11-03T16:31:18.6173759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6174138Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6174513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6174875Z return func(*args, **kwargs) 2025-11-03T16:31:18.6175234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6175599Z self_outputs = self.self( 2025-11-03T16:31:18.6175955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6176312Z return func(*args, **kwargs) 2025-11-03T16:31:18.6176708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6177073Z self.value(current_states) 2025-11-03T16:31:18.6177186Z 2025-11-03T16:31:18.6177266Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6177500Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6177845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6178150Z res = mod(**inputs) 2025-11-03T16:31:18.6178483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6178846Z outputs = self.bert( 2025-11-03T16:31:18.6179186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6179552Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6179907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6180262Z layer_outputs = layer_module( 2025-11-03T16:31:18.6180593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6180944Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6181307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6181665Z return func(*args, **kwargs) 2025-11-03T16:31:18.6182012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6182393Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6182765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6183142Z return func(*args, **kwargs) 2025-11-03T16:31:18.6183497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6183874Z self_outputs = self.self( 2025-11-03T16:31:18.6184233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6184637Z return func(*args, **kwargs) 2025-11-03T16:31:18.6185004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6185416Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6185600Z 2025-11-03T16:31:18.6185699Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6186052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6186366Z res = mod(**inputs) 2025-11-03T16:31:18.6186720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6187085Z outputs = self.bert( 2025-11-03T16:31:18.6187434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6187813Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6188180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6188548Z layer_outputs = layer_module( 2025-11-03T16:31:18.6188890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6189248Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6189622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6189988Z return func(*args, **kwargs) 2025-11-03T16:31:18.6190372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6190759Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6191140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6191507Z return func(*args, **kwargs) 2025-11-03T16:31:18.6191862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6192296Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6192724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6193111Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6193249Z 2025-11-03T16:31:18.6193358Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6193705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6194102Z res = mod(**inputs) 2025-11-03T16:31:18.6194464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6194852Z outputs = self.bert( 2025-11-03T16:31:18.6195203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6195579Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6195962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6196328Z layer_outputs = layer_module( 2025-11-03T16:31:18.6196660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6197000Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6197363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6197723Z return func(*args, **kwargs) 2025-11-03T16:31:18.6198130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6198507Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6198889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6199274Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6199668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6200117Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6200538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6200908Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6201049Z 2025-11-03T16:31:18.6201148Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6201502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6201815Z res = mod(**inputs) 2025-11-03T16:31:18.6202149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6202511Z outputs = self.bert( 2025-11-03T16:31:18.6202852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6203219Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6203580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6203942Z layer_outputs = layer_module( 2025-11-03T16:31:18.6204308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6204667Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6205029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6205388Z return func(*args, **kwargs) 2025-11-03T16:31:18.6205737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6206112Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6206505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6206888Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6207277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6207716Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6208135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6208534Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6208894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6209211Z return self.act(input) 2025-11-03T16:31:18.6209323Z 2025-11-03T16:31:18.6209421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6209759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6210060Z res = mod(**inputs) 2025-11-03T16:31:18.6210394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6210744Z outputs = self.bert( 2025-11-03T16:31:18.6211083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6211483Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6211838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6212187Z layer_outputs = layer_module( 2025-11-03T16:31:18.6212511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6212850Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6213305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6213669Z return func(*args, **kwargs) 2025-11-03T16:31:18.6214013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6214388Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6214770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6215146Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6215527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6215958Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6216368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6216735Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6216863Z 2025-11-03T16:31:18.6216973Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6217383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6217686Z res = mod(**inputs) 2025-11-03T16:31:18.6218024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6218382Z outputs = self.bert( 2025-11-03T16:31:18.6218718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6219071Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6219438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6219797Z layer_outputs = layer_module( 2025-11-03T16:31:18.6220127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6220472Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6220828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6221183Z return func(*args, **kwargs) 2025-11-03T16:31:18.6221534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6221902Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6222254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6222608Z return func(*args, **kwargs) 2025-11-03T16:31:18.6222954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6223315Z self_outputs = self.self( 2025-11-03T16:31:18.6223655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6223997Z return func(*args, **kwargs) 2025-11-03T16:31:18.6224343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6224901Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6225146Z 2025-11-03T16:31:18.6225252Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6225590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6225886Z res = mod(**inputs) 2025-11-03T16:31:18.6226221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6226579Z outputs = self.bert( 2025-11-03T16:31:18.6226916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6227273Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6227634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6227993Z layer_outputs = layer_module( 2025-11-03T16:31:18.6228325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6228668Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6229017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6229368Z return func(*args, **kwargs) 2025-11-03T16:31:18.6229713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6230080Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6230480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6230821Z return func(*args, **kwargs) 2025-11-03T16:31:18.6231171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6231531Z self_outputs = self.self( 2025-11-03T16:31:18.6231871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6232223Z return func(*args, **kwargs) 2025-11-03T16:31:18.6232577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6232943Z self.key(current_states) 2025-11-03T16:31:18.6233054Z 2025-11-03T16:31:18.6233162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6233510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6233823Z res = mod(**inputs) 2025-11-03T16:31:18.6234252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6234649Z outputs = self.bert( 2025-11-03T16:31:18.6235022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6235393Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6235750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6236121Z layer_outputs = layer_module( 2025-11-03T16:31:18.6236462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6236817Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6237182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6237548Z return func(*args, **kwargs) 2025-11-03T16:31:18.6237907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6238323Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6238696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6239048Z return func(*args, **kwargs) 2025-11-03T16:31:18.6239404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6239767Z self_outputs = self.self( 2025-11-03T16:31:18.6240117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6240470Z return func(*args, **kwargs) 2025-11-03T16:31:18.6240828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6241193Z self.value(current_states) 2025-11-03T16:31:18.6241307Z 2025-11-03T16:31:18.6241400Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6241634Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6241974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6242284Z res = mod(**inputs) 2025-11-03T16:31:18.6242627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6242990Z outputs = self.bert( 2025-11-03T16:31:18.6243337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6243707Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6244119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6244487Z layer_outputs = layer_module( 2025-11-03T16:31:18.6244826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6245170Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6245537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6245609Z return func(*args, **kwargs) 2025-11-03T16:31:18.6245839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6245917Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6246151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6246217Z return func(*args, **kwargs) 2025-11-03T16:31:18.6246456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6246523Z self_outputs = self.self( 2025-11-03T16:31:18.6246752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6246823Z return func(*args, **kwargs) 2025-11-03T16:31:18.6247054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6247183Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6247186Z 2025-11-03T16:31:18.6247286Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6247478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6247538Z res = mod(**inputs) 2025-11-03T16:31:18.6247774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6247842Z outputs = self.bert( 2025-11-03T16:31:18.6248071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6248188Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6248416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6248485Z layer_outputs = layer_module( 2025-11-03T16:31:18.6248700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6248774Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6249006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6249073Z return func(*args, **kwargs) 2025-11-03T16:31:18.6249306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6249391Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6249615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6249688Z return func(*args, **kwargs) 2025-11-03T16:31:18.6249915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6250046Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6250273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6250350Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6250353Z 2025-11-03T16:31:18.6250459Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6250673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6250742Z res = mod(**inputs) 2025-11-03T16:31:18.6250982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6251043Z outputs = self.bert( 2025-11-03T16:31:18.6251281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6251349Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6251587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6251653Z layer_outputs = layer_module( 2025-11-03T16:31:18.6251868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6251939Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6252167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6252243Z return func(*args, **kwargs) 2025-11-03T16:31:18.6252473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6252559Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6252803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6252876Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6253142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6253256Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6253495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6253572Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6253575Z 2025-11-03T16:31:18.6253710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6253895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6253956Z res = mod(**inputs) 2025-11-03T16:31:18.6254193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6254255Z outputs = self.bert( 2025-11-03T16:31:18.6254488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6254557Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6254783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6254862Z layer_outputs = layer_module( 2025-11-03T16:31:18.6255066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6255150Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6255372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6255436Z return func(*args, **kwargs) 2025-11-03T16:31:18.6255673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6255750Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6256000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6256072Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6256366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6256482Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6256718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6256830Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6257033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6257106Z return self.act(input) 2025-11-03T16:31:18.6257110Z 2025-11-03T16:31:18.6257207Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6257392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6257460Z res = mod(**inputs) 2025-11-03T16:31:18.6257696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6257764Z outputs = self.bert( 2025-11-03T16:31:18.6257997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6258077Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6258308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6258376Z layer_outputs = layer_module( 2025-11-03T16:31:18.6258589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6258662Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6258893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6258958Z return func(*args, **kwargs) 2025-11-03T16:31:18.6259191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6259276Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6259554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6259638Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6259912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6260043Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6260284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6260365Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6260368Z 2025-11-03T16:31:18.6260476Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6260667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6260738Z res = mod(**inputs) 2025-11-03T16:31:18.6260978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6261043Z outputs = self.bert( 2025-11-03T16:31:18.6261284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6261355Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6261595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6261665Z layer_outputs = layer_module( 2025-11-03T16:31:18.6261876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6261963Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6262228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6262306Z return func(*args, **kwargs) 2025-11-03T16:31:18.6262533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6262608Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6262840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6262905Z return func(*args, **kwargs) 2025-11-03T16:31:18.6263143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6263210Z self_outputs = self.self( 2025-11-03T16:31:18.6263442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6263510Z return func(*args, **kwargs) 2025-11-03T16:31:18.6263742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6263953Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6263956Z 2025-11-03T16:31:18.6264052Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6264244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6264304Z res = mod(**inputs) 2025-11-03T16:31:18.6264538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6264606Z outputs = self.bert( 2025-11-03T16:31:18.6264836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6264915Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6265142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6265270Z layer_outputs = layer_module( 2025-11-03T16:31:18.6265476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6265550Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6265780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6265844Z return func(*args, **kwargs) 2025-11-03T16:31:18.6266079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6266155Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6266381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6266453Z return func(*args, **kwargs) 2025-11-03T16:31:18.6266682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6266758Z self_outputs = self.self( 2025-11-03T16:31:18.6266982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6267045Z return func(*args, **kwargs) 2025-11-03T16:31:18.6267282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6267350Z self.key(current_states) 2025-11-03T16:31:18.6267353Z 2025-11-03T16:31:18.6267458Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6267641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6267749Z res = mod(**inputs) 2025-11-03T16:31:18.6267983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6268047Z outputs = self.bert( 2025-11-03T16:31:18.6268285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6268351Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6268585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6268650Z layer_outputs = layer_module( 2025-11-03T16:31:18.6268853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6268932Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6269157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6269230Z return func(*args, **kwargs) 2025-11-03T16:31:18.6269461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6269537Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6269771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6269835Z return func(*args, **kwargs) 2025-11-03T16:31:18.6270069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6270133Z self_outputs = self.self( 2025-11-03T16:31:18.6270362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6270424Z return func(*args, **kwargs) 2025-11-03T16:31:18.6270658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6270733Z self.value(current_states) 2025-11-03T16:31:18.6270765Z 2025-11-03T16:31:18.6270842Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6270947Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6271137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6271196Z res = mod(**inputs) 2025-11-03T16:31:18.6271435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6271495Z outputs = self.bert( 2025-11-03T16:31:18.6271738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6271805Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6272046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6272121Z layer_outputs = layer_module( 2025-11-03T16:31:18.6272331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6272415Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6272656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6272730Z return func(*args, **kwargs) 2025-11-03T16:31:18.6272968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6273045Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6273287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6273354Z return func(*args, **kwargs) 2025-11-03T16:31:18.6273628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6273698Z self_outputs = self.self( 2025-11-03T16:31:18.6273994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6274077Z return func(*args, **kwargs) 2025-11-03T16:31:18.6274336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6274472Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6274475Z 2025-11-03T16:31:18.6274574Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6274773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6274834Z res = mod(**inputs) 2025-11-03T16:31:18.6275086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6275160Z outputs = self.bert( 2025-11-03T16:31:18.6275407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6275487Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6275730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6275801Z layer_outputs = layer_module( 2025-11-03T16:31:18.6276031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6276109Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6276355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6276423Z return func(*args, **kwargs) 2025-11-03T16:31:18.6276681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6276813Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6277053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6277128Z return func(*args, **kwargs) 2025-11-03T16:31:18.6277370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6277504Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6277746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6277827Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6277830Z 2025-11-03T16:31:18.6277939Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6278138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6278210Z res = mod(**inputs) 2025-11-03T16:31:18.6278468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6278532Z outputs = self.bert( 2025-11-03T16:31:18.6278785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6278856Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6279111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6279182Z layer_outputs = layer_module( 2025-11-03T16:31:18.6279400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6279511Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6279749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6279828Z return func(*args, **kwargs) 2025-11-03T16:31:18.6280067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6280156Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6280413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6280488Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6280767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6280886Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6281139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6281221Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6281229Z 2025-11-03T16:31:18.6281331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6281532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6281595Z res = mod(**inputs) 2025-11-03T16:31:18.6281847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6281911Z outputs = self.bert( 2025-11-03T16:31:18.6282159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6282231Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6282476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6282555Z layer_outputs = layer_module( 2025-11-03T16:31:18.6282770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6282899Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6283138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6283206Z return func(*args, **kwargs) 2025-11-03T16:31:18.6283454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6283535Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6283801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6283878Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6284159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6284279Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6284520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6284636Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6284841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6284914Z return self.act(input) 2025-11-03T16:31:18.6284917Z 2025-11-03T16:31:18.6285014Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6285204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6285273Z res = mod(**inputs) 2025-11-03T16:31:18.6285547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6285619Z outputs = self.bert( 2025-11-03T16:31:18.6285855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6285927Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6286169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6286236Z layer_outputs = layer_module( 2025-11-03T16:31:18.6286455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6286530Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6286765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6286833Z return func(*args, **kwargs) 2025-11-03T16:31:18.6287069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6287157Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6287410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6287490Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6287754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6287881Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6288124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6288202Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6288206Z 2025-11-03T16:31:18.6288310Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6288505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6288574Z res = mod(**inputs) 2025-11-03T16:31:18.6288855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6288919Z outputs = self.bert( 2025-11-03T16:31:18.6289162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6289231Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6289474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6289542Z layer_outputs = layer_module( 2025-11-03T16:31:18.6289756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6289843Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6290075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6290154Z return func(*args, **kwargs) 2025-11-03T16:31:18.6290399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6290475Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6290708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6290774Z return func(*args, **kwargs) 2025-11-03T16:31:18.6291011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6291076Z self_outputs = self.self( 2025-11-03T16:31:18.6291348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6291415Z return func(*args, **kwargs) 2025-11-03T16:31:18.6291648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6291856Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6291859Z 2025-11-03T16:31:18.6291958Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6292152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6292213Z res = mod(**inputs) 2025-11-03T16:31:18.6292449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6292518Z outputs = self.bert( 2025-11-03T16:31:18.6292750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6292826Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6293059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6293135Z layer_outputs = layer_module( 2025-11-03T16:31:18.6293346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6293419Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6293657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6293722Z return func(*args, **kwargs) 2025-11-03T16:31:18.6293961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6294038Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6294267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6294340Z return func(*args, **kwargs) 2025-11-03T16:31:18.6294599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6294675Z self_outputs = self.self( 2025-11-03T16:31:18.6294900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6294968Z return func(*args, **kwargs) 2025-11-03T16:31:18.6295204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6295270Z self.key(current_states) 2025-11-03T16:31:18.6295273Z 2025-11-03T16:31:18.6295378Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6295559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6295631Z res = mod(**inputs) 2025-11-03T16:31:18.6295859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6295925Z outputs = self.bert( 2025-11-03T16:31:18.6296163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6296232Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6296467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6296532Z layer_outputs = layer_module( 2025-11-03T16:31:18.6296741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6296822Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6297069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6297141Z return func(*args, **kwargs) 2025-11-03T16:31:18.6297370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6297449Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6297680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6297744Z return func(*args, **kwargs) 2025-11-03T16:31:18.6297979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6298044Z self_outputs = self.self( 2025-11-03T16:31:18.6298280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6298344Z return func(*args, **kwargs) 2025-11-03T16:31:18.6298578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6298653Z self.value(current_states) 2025-11-03T16:31:18.6298659Z 2025-11-03T16:31:18.6298735Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6298837Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6299023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6299082Z res = mod(**inputs) 2025-11-03T16:31:18.6299323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6299386Z outputs = self.bert( 2025-11-03T16:31:18.6299622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6299689Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6299921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6299993Z layer_outputs = layer_module( 2025-11-03T16:31:18.6300247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6300329Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6300554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6300626Z return func(*args, **kwargs) 2025-11-03T16:31:18.6300852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6300927Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6301155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6301221Z return func(*args, **kwargs) 2025-11-03T16:31:18.6301459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6301525Z self_outputs = self.self( 2025-11-03T16:31:18.6301749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6301821Z return func(*args, **kwargs) 2025-11-03T16:31:18.6302046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6302178Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6302181Z 2025-11-03T16:31:18.6302276Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6302461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6302529Z res = mod(**inputs) 2025-11-03T16:31:18.6302798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6302870Z outputs = self.bert( 2025-11-03T16:31:18.6303106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6303182Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6303417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6303484Z layer_outputs = layer_module( 2025-11-03T16:31:18.6303703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6303775Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6304010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6304075Z return func(*args, **kwargs) 2025-11-03T16:31:18.6304308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6304397Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6304624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6304695Z return func(*args, **kwargs) 2025-11-03T16:31:18.6304927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6305048Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6305290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6305366Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6305369Z 2025-11-03T16:31:18.6305475Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6305659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6305759Z res = mod(**inputs) 2025-11-03T16:31:18.6305991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6306052Z outputs = self.bert( 2025-11-03T16:31:18.6306288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6306355Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6306590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6306655Z layer_outputs = layer_module( 2025-11-03T16:31:18.6306860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6306941Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6307163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6307238Z return func(*args, **kwargs) 2025-11-03T16:31:18.6307464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6307549Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6307797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6307871Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6308139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6308251Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6308520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6308599Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6308605Z 2025-11-03T16:31:18.6308700Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6308891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6308951Z res = mod(**inputs) 2025-11-03T16:31:18.6309188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6309248Z outputs = self.bert( 2025-11-03T16:31:18.6309481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6309550Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6309778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6309853Z layer_outputs = layer_module( 2025-11-03T16:31:18.6310057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6310141Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6310371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6310438Z return func(*args, **kwargs) 2025-11-03T16:31:18.6310677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6310756Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6311012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6311085Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6311348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6311510Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6311747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6311864Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6312093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6312168Z return self.act(input) 2025-11-03T16:31:18.6312171Z 2025-11-03T16:31:18.6312271Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6312459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6312526Z res = mod(**inputs) 2025-11-03T16:31:18.6312772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6312841Z outputs = self.bert( 2025-11-03T16:31:18.6313077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6313148Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6313511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6313585Z layer_outputs = layer_module( 2025-11-03T16:31:18.6313806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6313880Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6314208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6314285Z return func(*args, **kwargs) 2025-11-03T16:31:18.6314591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6314688Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6314939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6315019Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6315283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6315410Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6315652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6315731Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6315735Z 2025-11-03T16:31:18.6315844Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6316034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6316099Z res = mod(**inputs) 2025-11-03T16:31:18.6316345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6316411Z outputs = self.bert( 2025-11-03T16:31:18.6316653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6316723Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6316969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6317041Z layer_outputs = layer_module( 2025-11-03T16:31:18.6317255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6317352Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6317583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6317704Z return func(*args, **kwargs) 2025-11-03T16:31:18.6317939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6318017Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6318257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6318324Z return func(*args, **kwargs) 2025-11-03T16:31:18.6318575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6318643Z self_outputs = self.self( 2025-11-03T16:31:18.6318876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6318949Z return func(*args, **kwargs) 2025-11-03T16:31:18.6319186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-11-03T16:31:18.6319396Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:31:18.6319399Z 2025-11-03T16:31:18.6319500Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6319698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6319762Z res = mod(**inputs) 2025-11-03T16:31:18.6320000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6320071Z outputs = self.bert( 2025-11-03T16:31:18.6320338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6320418Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6320653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6320723Z layer_outputs = layer_module( 2025-11-03T16:31:18.6320942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6321016Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6321252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6321318Z return func(*args, **kwargs) 2025-11-03T16:31:18.6321562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6321641Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6321872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6321944Z return func(*args, **kwargs) 2025-11-03T16:31:18.6322183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6322255Z self_outputs = self.self( 2025-11-03T16:31:18.6322488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6322552Z return func(*args, **kwargs) 2025-11-03T16:31:18.6322800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 386, in forward 2025-11-03T16:31:18.6322868Z self.key(current_states) 2025-11-03T16:31:18.6322871Z 2025-11-03T16:31:18.6322976Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6323168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6323230Z res = mod(**inputs) 2025-11-03T16:31:18.6323473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6323569Z outputs = self.bert( 2025-11-03T16:31:18.6323806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6323876Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6324117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6324184Z layer_outputs = layer_module( 2025-11-03T16:31:18.6324393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6324477Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6324707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6324780Z return func(*args, **kwargs) 2025-11-03T16:31:18.6325015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6325106Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6325335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6325398Z return func(*args, **kwargs) 2025-11-03T16:31:18.6325632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6325696Z self_outputs = self.self( 2025-11-03T16:31:18.6325916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6325989Z return func(*args, **kwargs) 2025-11-03T16:31:18.6326245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 391, in forward 2025-11-03T16:31:18.6326323Z self.value(current_states) 2025-11-03T16:31:18.6326326Z 2025-11-03T16:31:18.6326403Z cudagraph partition due to non gpu ops 2025-11-03T16:31:18.6326508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6326692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6326753Z res = mod(**inputs) 2025-11-03T16:31:18.6326994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6327055Z outputs = self.bert( 2025-11-03T16:31:18.6327289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6327356Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6327585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6327659Z layer_outputs = layer_module( 2025-11-03T16:31:18.6327868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6327948Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6328170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6328233Z return func(*args, **kwargs) 2025-11-03T16:31:18.6328467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6328543Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6328772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6328839Z return func(*args, **kwargs) 2025-11-03T16:31:18.6329081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 487, in forward 2025-11-03T16:31:18.6329177Z self_outputs = self.self( 2025-11-03T16:31:18.6329401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6329472Z return func(*args, **kwargs) 2025-11-03T16:31:18.6329698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 412, in forward 2025-11-03T16:31:18.6329830Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:31:18.6329834Z 2025-11-03T16:31:18.6329929Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6330111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6330177Z res = mod(**inputs) 2025-11-03T16:31:18.6330411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6330482Z outputs = self.bert( 2025-11-03T16:31:18.6330711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6330778Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6331017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6331083Z layer_outputs = layer_module( 2025-11-03T16:31:18.6331295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6331369Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6331598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6331694Z return func(*args, **kwargs) 2025-11-03T16:31:18.6331927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 557, in forward 2025-11-03T16:31:18.6332015Z self_attention_outputs = self.attention( 2025-11-03T16:31:18.6332237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6332321Z return func(*args, **kwargs) 2025-11-03T16:31:18.6332549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward 2025-11-03T16:31:18.6332671Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:31:18.6332907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 435, in forward 2025-11-03T16:31:18.6332985Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6332988Z 2025-11-03T16:31:18.6333097Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6333278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6333346Z res = mod(**inputs) 2025-11-03T16:31:18.6333579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6333641Z outputs = self.bert( 2025-11-03T16:31:18.6333873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6333941Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6334178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6334244Z layer_outputs = layer_module( 2025-11-03T16:31:18.6334450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6334532Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6334753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6334865Z return func(*args, **kwargs) 2025-11-03T16:31:18.6335092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6335172Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6335422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6335493Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6335758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6335870Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6336108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 511, in forward 2025-11-03T16:31:18.6336187Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6336190Z 2025-11-03T16:31:18.6336287Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6336477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6336538Z res = mod(**inputs) 2025-11-03T16:31:18.6336777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6336837Z outputs = self.bert( 2025-11-03T16:31:18.6337070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6337144Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6337404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6337481Z layer_outputs = layer_module( 2025-11-03T16:31:18.6337686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6337765Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6337990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6338053Z return func(*args, **kwargs) 2025-11-03T16:31:18.6338288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6338365Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6338613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6338685Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6338944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 595, in feed_forward_chunk 2025-11-03T16:31:18.6339065Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:31:18.6339296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-11-03T16:31:18.6339409Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:31:18.6339609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:31:18.6339681Z return self.act(input) 2025-11-03T16:31:18.6339685Z 2025-11-03T16:31:18.6339780Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6339963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6340030Z res = mod(**inputs) 2025-11-03T16:31:18.6340263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1279, in forward 2025-11-03T16:31:18.6340329Z outputs = self.bert( 2025-11-03T16:31:18.6340589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 999, in forward 2025-11-03T16:31:18.6340657Z encoder_outputs = self.encoder( 2025-11-03T16:31:18.6340891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 649, in forward 2025-11-03T16:31:18.6340956Z layer_outputs = layer_module( 2025-11-03T16:31:18.6341168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:31:18.6341239Z return super().__call__(*args, **kwargs) 2025-11-03T16:31:18.6341463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:31:18.6341537Z return func(*args, **kwargs) 2025-11-03T16:31:18.6341763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 587, in forward 2025-11-03T16:31:18.6341851Z layer_output = apply_chunking_to_forward( 2025-11-03T16:31:18.6342093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:31:18.6342171Z return forward_fn(*input_tensors) 2025-11-03T16:31:18.6342429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-11-03T16:31:18.6342554Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:31:18.6342792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-11-03T16:31:18.6342868Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6342871Z 2025-11-03T16:31:18.6343005Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6343193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6343260Z res = mod(**inputs) 2025-11-03T16:31:18.6343508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1294, in forward 2025-11-03T16:31:18.6343602Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:31:18.6343845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 751, in forward 2025-11-03T16:31:18.6343953Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:31:18.6344200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 740, in forward 2025-11-03T16:31:18.6344288Z hidden_states = self.transform(hidden_states) 2025-11-03T16:31:18.6344532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 716, in forward 2025-11-03T16:31:18.6344618Z hidden_states = self.dense(hidden_states) 2025-11-03T16:31:18.6344624Z 2025-11-03T16:31:18.6344721Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6344917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6344980Z res = mod(**inputs) 2025-11-03T16:31:18.6345219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1294, in forward 2025-11-03T16:31:18.6345317Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:31:18.6345553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 751, in forward 2025-11-03T16:31:18.6345683Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:31:18.6345922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 741, in forward 2025-11-03T16:31:18.6346021Z hidden_states = self.decoder(hidden_states) 2025-11-03T16:31:18.6346058Z 2025-11-03T16:31:18.6346157Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:31:18.6346351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:31:18.6346421Z res = mod(**inputs) 2025-11-03T16:31:18.6346658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1299, in forward 2025-11-03T16:31:18.6346844Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:31:18.6346847Z 2025-11-03T16:31:28.3026229Z Compilation time (from dynamo_timed): 15.648786153 2025-11-03T16:31:28.3100064Z pass 2025-11-03T16:31:28.3104803Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:31:28.3108702Z TIMING: _recursive_pre_grad_passes:0.00611 _recursive_joint_graph_passes:0.62126 _recursive_post_grad_passes:0.06067 async_compile.wait:0.67903 code_gen:8.62289 inductor_compile:9.86019 backend_compile:12.78861 gc:0.00043 entire_frame_compile:15.64879 total_wall_time:15.64879 2025-11-03T16:31:28.3109628Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:7049 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:1946 2025-11-03T16:31:28.3110106Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-11-03T16:31:30.3969714Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:31:30.3970609Z import pynvml # type: ignore[import] 2025-11-03T16:31:33.5001403Z 2025-11-03T16:31:48.7623718Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:31:48.7627979Z loading model: 0it [00:15, ?it/s] 2025-11-03T16:31:48.7651300Z cpu eval BlenderbotForCausalLM 2025-11-03T16:31:48.7827397Z Compilation time (from dynamo_timed): 0 2025-11-03T16:31:48.7829052Z pass_due_to_skip 2025-11-03T16:31:48.7829498Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:31:48.7834064Z TIMING: total_wall_time:0 2025-11-03T16:31:48.7838546Z STATS: call_* op count: 0 2025-11-03T16:31:48.7840529Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-11-03T16:31:50.6465367Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:31:50.6466179Z import pynvml # type: ignore[import] 2025-11-03T16:31:53.8134678Z 2025-11-03T16:32:00.9076604Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:32:00.9081224Z loading model: 0it [00:07, ?it/s] 2025-11-03T16:32:00.9102679Z cpu eval DebertaV2ForMaskedLM 2025-11-03T16:32:00.9276660Z Compilation time (from dynamo_timed): 0 2025-11-03T16:32:00.9278466Z pass_due_to_skip 2025-11-03T16:32:00.9278911Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:00.9283454Z TIMING: total_wall_time:0 2025-11-03T16:32:00.9285510Z STATS: call_* op count: 0 2025-11-03T16:32:00.9285895Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-11-03T16:32:02.7411691Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:32:02.7412731Z import pynvml # type: ignore[import] 2025-11-03T16:32:05.9128816Z 2025-11-03T16:32:06.5375734Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:32:06.5376232Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:32:06.5383283Z cpu eval DistilBertForMaskedLM 2025-11-03T16:32:06.6955438Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:06.7459133Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:06.7922634Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:10.9128179Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9129770Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9130129Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9136458Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9139546Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9144608Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9152000Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9152588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9153077Z res = mod(**inputs) 2025-11-03T16:32:10.9153540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9154067Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9154581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9159324Z return self.transformer( 2025-11-03T16:32:10.9163293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9167318Z layer_outputs = layer_module( 2025-11-03T16:32:10.9171979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9176139Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9180478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9182045Z sa_output = self.attention( 2025-11-03T16:32:10.9182483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-11-03T16:32:10.9183060Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-11-03T16:32:10.9183244Z 2025-11-03T16:32:10.9183364Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9183735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9184075Z res = mod(**inputs) 2025-11-03T16:32:10.9184500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9184929Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9185340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9185749Z return self.transformer( 2025-11-03T16:32:10.9186139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9186546Z layer_outputs = layer_module( 2025-11-03T16:32:10.9186925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9187313Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9187752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9188218Z sa_output = self.attention( 2025-11-03T16:32:10.9188630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-11-03T16:32:10.9189367Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9189562Z 2025-11-03T16:32:10.9189672Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9190067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9190386Z res = mod(**inputs) 2025-11-03T16:32:10.9190753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9191154Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9191551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9191953Z return self.transformer( 2025-11-03T16:32:10.9192347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9192749Z layer_outputs = layer_module( 2025-11-03T16:32:10.9193097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9193462Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9193876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9194406Z sa_output = self.attention( 2025-11-03T16:32:10.9194820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-11-03T16:32:10.9195296Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9195474Z 2025-11-03T16:32:10.9195647Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9195895Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9196241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9196558Z res = mod(**inputs) 2025-11-03T16:32:10.9196941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9197353Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9197765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9198168Z return self.transformer( 2025-11-03T16:32:10.9198569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9198979Z layer_outputs = layer_module( 2025-11-03T16:32:10.9199319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9199666Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9200072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9200469Z sa_output = self.attention( 2025-11-03T16:32:10.9200856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-11-03T16:32:10.9201312Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:10.9201490Z 2025-11-03T16:32:10.9201593Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9201939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9202256Z res = mod(**inputs) 2025-11-03T16:32:10.9202635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9203036Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9203485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9203879Z return self.transformer( 2025-11-03T16:32:10.9204266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9204717Z layer_outputs = layer_module( 2025-11-03T16:32:10.9205046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9205400Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9205806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9206202Z sa_output = self.attention( 2025-11-03T16:32:10.9206588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-11-03T16:32:10.9206986Z attn_output = self.out_lin(attn_output) 2025-11-03T16:32:10.9207130Z 2025-11-03T16:32:10.9207231Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9207576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9207890Z res = mod(**inputs) 2025-11-03T16:32:10.9208259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9208650Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9209040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9209469Z return self.transformer( 2025-11-03T16:32:10.9209854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9210250Z layer_outputs = layer_module( 2025-11-03T16:32:10.9210577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9210927Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9211326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9211761Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9212182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9212706Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9213415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9213825Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9214236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-11-03T16:32:10.9214632Z x = self.lin1(input) 2025-11-03T16:32:10.9214748Z 2025-11-03T16:32:10.9214851Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9215206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9215522Z res = mod(**inputs) 2025-11-03T16:32:10.9215894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9216290Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9216694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9217137Z return self.transformer( 2025-11-03T16:32:10.9217520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9217919Z layer_outputs = layer_module( 2025-11-03T16:32:10.9218249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9218591Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9218990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9219421Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9219851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9220373Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9220879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9221267Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9221655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-11-03T16:32:10.9222033Z x = self.activation(x) 2025-11-03T16:32:10.9222344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:10.9222673Z return self.act(input) 2025-11-03T16:32:10.9222782Z 2025-11-03T16:32:10.9222893Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9223291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9223586Z res = mod(**inputs) 2025-11-03T16:32:10.9223949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9224340Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9224731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9225127Z return self.transformer( 2025-11-03T16:32:10.9225501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9225891Z layer_outputs = layer_module( 2025-11-03T16:32:10.9226240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9226591Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9226985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9227419Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9227847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9228361Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9228855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9229225Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9229620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-11-03T16:32:10.9230006Z x = self.lin2(x) 2025-11-03T16:32:10.9230101Z 2025-11-03T16:32:10.9230211Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9230559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9230899Z res = mod(**inputs) 2025-11-03T16:32:10.9231284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9231689Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9232091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9232491Z return self.transformer( 2025-11-03T16:32:10.9232876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9233274Z layer_outputs = layer_module( 2025-11-03T16:32:10.9233624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9234047Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9234514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9234929Z sa_output = self.attention( 2025-11-03T16:32:10.9235335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-11-03T16:32:10.9235895Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-11-03T16:32:10.9236072Z 2025-11-03T16:32:10.9236182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9236527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9236846Z res = mod(**inputs) 2025-11-03T16:32:10.9237261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9237676Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9238084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9238494Z return self.transformer( 2025-11-03T16:32:10.9238887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9239293Z layer_outputs = layer_module( 2025-11-03T16:32:10.9239635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9239987Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9240395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9240801Z sa_output = self.attention( 2025-11-03T16:32:10.9241206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-11-03T16:32:10.9241661Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9241831Z 2025-11-03T16:32:10.9241933Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9242286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9242601Z res = mod(**inputs) 2025-11-03T16:32:10.9242973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9243374Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9243769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9244181Z return self.transformer( 2025-11-03T16:32:10.9244561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9244976Z layer_outputs = layer_module( 2025-11-03T16:32:10.9245292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9245636Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9246025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9246405Z sa_output = self.attention( 2025-11-03T16:32:10.9246781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-11-03T16:32:10.9247200Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9247374Z 2025-11-03T16:32:10.9247455Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9247684Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9248019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9248323Z res = mod(**inputs) 2025-11-03T16:32:10.9248676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9249059Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9249443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9249826Z return self.transformer( 2025-11-03T16:32:10.9250192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9250572Z layer_outputs = layer_module( 2025-11-03T16:32:10.9250955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9251296Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9251681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9252054Z sa_output = self.attention( 2025-11-03T16:32:10.9252424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-11-03T16:32:10.9252861Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:10.9253034Z 2025-11-03T16:32:10.9253138Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9253472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9253763Z res = mod(**inputs) 2025-11-03T16:32:10.9254128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9254513Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9254899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9255280Z return self.transformer( 2025-11-03T16:32:10.9255645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9256024Z layer_outputs = layer_module( 2025-11-03T16:32:10.9256348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9256685Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9257065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9257449Z sa_output = self.attention( 2025-11-03T16:32:10.9257823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-11-03T16:32:10.9258286Z attn_output = self.out_lin(attn_output) 2025-11-03T16:32:10.9258415Z 2025-11-03T16:32:10.9258520Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9258845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9259151Z res = mod(**inputs) 2025-11-03T16:32:10.9259507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9259891Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9260267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9260647Z return self.transformer( 2025-11-03T16:32:10.9261017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9261402Z layer_outputs = layer_module( 2025-11-03T16:32:10.9261727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9262061Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9262445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9262862Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9263286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9263783Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9264289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9264662Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9265062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-11-03T16:32:10.9265454Z x = self.lin1(input) 2025-11-03T16:32:10.9265556Z 2025-11-03T16:32:10.9265663Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9266002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9266311Z res = mod(**inputs) 2025-11-03T16:32:10.9266678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9267071Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9267462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9267846Z return self.transformer( 2025-11-03T16:32:10.9268226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9268620Z layer_outputs = layer_module( 2025-11-03T16:32:10.9268953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9269292Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9269687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9270114Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9270549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9271061Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9271582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9271970Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9272369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-11-03T16:32:10.9272765Z x = self.activation(x) 2025-11-03T16:32:10.9273088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:10.9273415Z return self.act(input) 2025-11-03T16:32:10.9273531Z 2025-11-03T16:32:10.9273631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9274067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9274404Z res = mod(**inputs) 2025-11-03T16:32:10.9274792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9275205Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9275628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9276033Z return self.transformer( 2025-11-03T16:32:10.9276427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9276822Z layer_outputs = layer_module( 2025-11-03T16:32:10.9277168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9277526Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9277976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9278407Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9278832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9279346Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9279843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9291961Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9292436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-11-03T16:32:10.9292840Z x = self.lin2(x) 2025-11-03T16:32:10.9292956Z 2025-11-03T16:32:10.9293079Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9293439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9293770Z res = mod(**inputs) 2025-11-03T16:32:10.9294147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9294557Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9294960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9295366Z return self.transformer( 2025-11-03T16:32:10.9295751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9296143Z layer_outputs = layer_module( 2025-11-03T16:32:10.9296486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9296843Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9297245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9297738Z sa_output = self.attention( 2025-11-03T16:32:10.9298109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-11-03T16:32:10.9298546Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-11-03T16:32:10.9298728Z 2025-11-03T16:32:10.9298832Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9299178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9299477Z res = mod(**inputs) 2025-11-03T16:32:10.9299850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9300240Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9300630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9301019Z return self.transformer( 2025-11-03T16:32:10.9301386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9301772Z layer_outputs = layer_module( 2025-11-03T16:32:10.9302103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9302448Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9302837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9303214Z sa_output = self.attention( 2025-11-03T16:32:10.9303629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-11-03T16:32:10.9304068Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9304234Z 2025-11-03T16:32:10.9304345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9304687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9304987Z res = mod(**inputs) 2025-11-03T16:32:10.9305352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9305744Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9306132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9306513Z return self.transformer( 2025-11-03T16:32:10.9306895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9307387Z layer_outputs = layer_module( 2025-11-03T16:32:10.9307725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9308071Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9308459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9308846Z sa_output = self.attention( 2025-11-03T16:32:10.9309226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-11-03T16:32:10.9309660Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9309829Z 2025-11-03T16:32:10.9309914Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9310138Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9310481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9310821Z res = mod(**inputs) 2025-11-03T16:32:10.9311184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9311561Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9311944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9312350Z return self.transformer( 2025-11-03T16:32:10.9312734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9313131Z layer_outputs = layer_module( 2025-11-03T16:32:10.9313673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9314094Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9314503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9314902Z sa_output = self.attention( 2025-11-03T16:32:10.9315339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-11-03T16:32:10.9315790Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:10.9315978Z 2025-11-03T16:32:10.9316082Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9316427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9316742Z res = mod(**inputs) 2025-11-03T16:32:10.9317211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9317609Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9318008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9318408Z return self.transformer( 2025-11-03T16:32:10.9318804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9319193Z layer_outputs = layer_module( 2025-11-03T16:32:10.9319528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9319882Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9320287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9320687Z sa_output = self.attention( 2025-11-03T16:32:10.9321069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-11-03T16:32:10.9321482Z attn_output = self.out_lin(attn_output) 2025-11-03T16:32:10.9321624Z 2025-11-03T16:32:10.9321725Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9322073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9322386Z res = mod(**inputs) 2025-11-03T16:32:10.9322746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9323143Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9323537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9323934Z return self.transformer( 2025-11-03T16:32:10.9324314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9324759Z layer_outputs = layer_module( 2025-11-03T16:32:10.9325095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9325456Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9325856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9326286Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9326717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9327239Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9327730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9328104Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9328489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-11-03T16:32:10.9328876Z x = self.lin1(input) 2025-11-03T16:32:10.9328985Z 2025-11-03T16:32:10.9329083Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9329422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9329726Z res = mod(**inputs) 2025-11-03T16:32:10.9330078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9330466Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9330876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9331264Z return self.transformer( 2025-11-03T16:32:10.9331636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9332030Z layer_outputs = layer_module( 2025-11-03T16:32:10.9332362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9332708Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9333105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9333518Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9333941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9334454Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9334944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9335322Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9335705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-11-03T16:32:10.9336090Z x = self.activation(x) 2025-11-03T16:32:10.9336399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:10.9336722Z return self.act(input) 2025-11-03T16:32:10.9336827Z 2025-11-03T16:32:10.9336935Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9337267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9337576Z res = mod(**inputs) 2025-11-03T16:32:10.9337939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9338385Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9338759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9339142Z return self.transformer( 2025-11-03T16:32:10.9339514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9339902Z layer_outputs = layer_module( 2025-11-03T16:32:10.9340224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9340557Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9340959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9341375Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9341798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9342300Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9342784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9343155Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9343541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-11-03T16:32:10.9343918Z x = self.lin2(x) 2025-11-03T16:32:10.9344012Z 2025-11-03T16:32:10.9344149Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9344488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9344808Z res = mod(**inputs) 2025-11-03T16:32:10.9345179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9345575Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9345959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9346352Z return self.transformer( 2025-11-03T16:32:10.9346730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9347123Z layer_outputs = layer_module( 2025-11-03T16:32:10.9347468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9347808Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9348206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9348601Z sa_output = self.attention( 2025-11-03T16:32:10.9348985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-11-03T16:32:10.9349429Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-11-03T16:32:10.9349600Z 2025-11-03T16:32:10.9349704Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9350049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9350360Z res = mod(**inputs) 2025-11-03T16:32:10.9350731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9351119Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9351515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9351937Z return self.transformer( 2025-11-03T16:32:10.9352312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9352690Z layer_outputs = layer_module( 2025-11-03T16:32:10.9353022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9353367Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9353760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9354221Z sa_output = self.attention( 2025-11-03T16:32:10.9354601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-11-03T16:32:10.9355049Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9355230Z 2025-11-03T16:32:10.9355331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9355681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9355997Z res = mod(**inputs) 2025-11-03T16:32:10.9356361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9356765Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9357186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9357583Z return self.transformer( 2025-11-03T16:32:10.9357991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9358386Z layer_outputs = layer_module( 2025-11-03T16:32:10.9358725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9359075Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9359476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9359858Z sa_output = self.attention( 2025-11-03T16:32:10.9360241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-11-03T16:32:10.9360685Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9360856Z 2025-11-03T16:32:10.9360943Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9361176Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9361511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9361826Z res = mod(**inputs) 2025-11-03T16:32:10.9362200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9362595Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9362983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9363372Z return self.transformer( 2025-11-03T16:32:10.9363748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9364143Z layer_outputs = layer_module( 2025-11-03T16:32:10.9364477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9364821Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9365232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9365674Z sa_output = self.attention( 2025-11-03T16:32:10.9366059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-11-03T16:32:10.9366511Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:10.9366689Z 2025-11-03T16:32:10.9366788Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9367133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9367446Z res = mod(**inputs) 2025-11-03T16:32:10.9367816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9368204Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9368603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9368992Z return self.transformer( 2025-11-03T16:32:10.9369366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9369752Z layer_outputs = layer_module( 2025-11-03T16:32:10.9370073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9370419Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9370812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9371199Z sa_output = self.attention( 2025-11-03T16:32:10.9371602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-11-03T16:32:10.9371992Z attn_output = self.out_lin(attn_output) 2025-11-03T16:32:10.9372129Z 2025-11-03T16:32:10.9372226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9372561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9372863Z res = mod(**inputs) 2025-11-03T16:32:10.9373212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9373597Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9373980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9374361Z return self.transformer( 2025-11-03T16:32:10.9374735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9375108Z layer_outputs = layer_module( 2025-11-03T16:32:10.9375435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9375772Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9376161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9376581Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9376992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9377497Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9377986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9378361Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9378788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-11-03T16:32:10.9379166Z x = self.lin1(input) 2025-11-03T16:32:10.9379274Z 2025-11-03T16:32:10.9379370Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9379707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9380009Z res = mod(**inputs) 2025-11-03T16:32:10.9380360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9380744Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9381123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9381507Z return self.transformer( 2025-11-03T16:32:10.9381878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9382255Z layer_outputs = layer_module( 2025-11-03T16:32:10.9382578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9382920Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9383315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9383744Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9384174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9384710Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9385196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9385575Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9385976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-11-03T16:32:10.9386358Z x = self.activation(x) 2025-11-03T16:32:10.9386677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:10.9387007Z return self.act(input) 2025-11-03T16:32:10.9387113Z 2025-11-03T16:32:10.9387222Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9387566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9387870Z res = mod(**inputs) 2025-11-03T16:32:10.9388237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9388633Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9389022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9393420Z return self.transformer( 2025-11-03T16:32:10.9393808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9394270Z layer_outputs = layer_module( 2025-11-03T16:32:10.9394618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9394979Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9395376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9395795Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9396255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9396754Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9397255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9397629Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9398021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-11-03T16:32:10.9398401Z x = self.lin2(x) 2025-11-03T16:32:10.9398496Z 2025-11-03T16:32:10.9398593Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9398938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9399247Z res = mod(**inputs) 2025-11-03T16:32:10.9399607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9399990Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9400364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9400746Z return self.transformer( 2025-11-03T16:32:10.9401113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9401493Z layer_outputs = layer_module( 2025-11-03T16:32:10.9401812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9402188Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9402578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9402962Z sa_output = self.attention( 2025-11-03T16:32:10.9403331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-11-03T16:32:10.9403755Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-11-03T16:32:10.9403930Z 2025-11-03T16:32:10.9404026Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9404356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9404658Z res = mod(**inputs) 2025-11-03T16:32:10.9405014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9405390Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9405780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9406166Z return self.transformer( 2025-11-03T16:32:10.9406534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9406975Z layer_outputs = layer_module( 2025-11-03T16:32:10.9407297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9407644Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9408037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9408426Z sa_output = self.attention( 2025-11-03T16:32:10.9408802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-11-03T16:32:10.9409239Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9409428Z 2025-11-03T16:32:10.9409527Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9409874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9410185Z res = mod(**inputs) 2025-11-03T16:32:10.9410546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9410942Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9411335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9411727Z return self.transformer( 2025-11-03T16:32:10.9412110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9412495Z layer_outputs = layer_module( 2025-11-03T16:32:10.9412830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9413177Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9413736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9414127Z sa_output = self.attention( 2025-11-03T16:32:10.9414505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-11-03T16:32:10.9414947Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9415116Z 2025-11-03T16:32:10.9415203Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9415485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9415824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9416125Z res = mod(**inputs) 2025-11-03T16:32:10.9416484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9416860Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9417241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9417623Z return self.transformer( 2025-11-03T16:32:10.9417994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9418375Z layer_outputs = layer_module( 2025-11-03T16:32:10.9418694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9419037Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9419426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9419808Z sa_output = self.attention( 2025-11-03T16:32:10.9420172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-11-03T16:32:10.9420665Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:10.9420844Z 2025-11-03T16:32:10.9420941Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9421275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9421576Z res = mod(**inputs) 2025-11-03T16:32:10.9421949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9422327Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9422706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9423114Z return self.transformer( 2025-11-03T16:32:10.9423484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9423869Z layer_outputs = layer_module( 2025-11-03T16:32:10.9424186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9424525Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9424913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9425295Z sa_output = self.attention( 2025-11-03T16:32:10.9425671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-11-03T16:32:10.9426055Z attn_output = self.out_lin(attn_output) 2025-11-03T16:32:10.9426194Z 2025-11-03T16:32:10.9426291Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9426626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9426929Z res = mod(**inputs) 2025-11-03T16:32:10.9427287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9427669Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9428051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9428432Z return self.transformer( 2025-11-03T16:32:10.9428835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9429209Z layer_outputs = layer_module( 2025-11-03T16:32:10.9429531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9429878Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9430266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9430687Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9431095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9431591Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9432075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9432444Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9432827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-11-03T16:32:10.9433198Z x = self.lin1(input) 2025-11-03T16:32:10.9433304Z 2025-11-03T16:32:10.9433398Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9433759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9434123Z res = mod(**inputs) 2025-11-03T16:32:10.9434487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9434882Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9435275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9435675Z return self.transformer( 2025-11-03T16:32:10.9436051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9436458Z layer_outputs = layer_module( 2025-11-03T16:32:10.9436797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9437154Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9437554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9437992Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9438419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9438937Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9439436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9439824Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9440221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-11-03T16:32:10.9440613Z x = self.activation(x) 2025-11-03T16:32:10.9440929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:10.9441260Z return self.act(input) 2025-11-03T16:32:10.9441365Z 2025-11-03T16:32:10.9441474Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9441812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9442123Z res = mod(**inputs) 2025-11-03T16:32:10.9442526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9442924Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9443316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9443700Z return self.transformer( 2025-11-03T16:32:10.9444085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9444477Z layer_outputs = layer_module( 2025-11-03T16:32:10.9444813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9445167Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9445564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9445993Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9446419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9446934Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9447453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9447833Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9448234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-11-03T16:32:10.9448632Z x = self.lin2(x) 2025-11-03T16:32:10.9448731Z 2025-11-03T16:32:10.9448841Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9449187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9449496Z res = mod(**inputs) 2025-11-03T16:32:10.9449869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9450251Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9450628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9451003Z return self.transformer( 2025-11-03T16:32:10.9451374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9451757Z layer_outputs = layer_module( 2025-11-03T16:32:10.9452084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9452428Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9452811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9453195Z sa_output = self.attention( 2025-11-03T16:32:10.9453571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-11-03T16:32:10.9454010Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-11-03T16:32:10.9454178Z 2025-11-03T16:32:10.9454284Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9454612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9454915Z res = mod(**inputs) 2025-11-03T16:32:10.9455273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9455688Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9456064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9456452Z return self.transformer( 2025-11-03T16:32:10.9456825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9457213Z layer_outputs = layer_module( 2025-11-03T16:32:10.9457538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9457873Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9458262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9458650Z sa_output = self.attention( 2025-11-03T16:32:10.9459027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-11-03T16:32:10.9459458Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9459624Z 2025-11-03T16:32:10.9459720Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9460055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9460359Z res = mod(**inputs) 2025-11-03T16:32:10.9460749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9461125Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9461519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9461917Z return self.transformer( 2025-11-03T16:32:10.9462298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9462690Z layer_outputs = layer_module( 2025-11-03T16:32:10.9463011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9463379Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9463779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9464176Z sa_output = self.attention( 2025-11-03T16:32:10.9464560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-11-03T16:32:10.9464994Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-11-03T16:32:10.9465172Z 2025-11-03T16:32:10.9465251Z cudagraph partition due to non gpu ops 2025-11-03T16:32:10.9465479Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9465832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9466131Z res = mod(**inputs) 2025-11-03T16:32:10.9466493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9466876Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9467255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9467640Z return self.transformer( 2025-11-03T16:32:10.9468001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9468380Z layer_outputs = layer_module( 2025-11-03T16:32:10.9468702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9469275Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9469676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9470056Z sa_output = self.attention( 2025-11-03T16:32:10.9470429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-11-03T16:32:10.9470871Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:10.9471044Z 2025-11-03T16:32:10.9471147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9471483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9471775Z res = mod(**inputs) 2025-11-03T16:32:10.9472141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9472539Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9472931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9473321Z return self.transformer( 2025-11-03T16:32:10.9473700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9474243Z layer_outputs = layer_module( 2025-11-03T16:32:10.9474590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9474955Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9475362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-11-03T16:32:10.9475783Z sa_output = self.attention( 2025-11-03T16:32:10.9476173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-11-03T16:32:10.9476578Z attn_output = self.out_lin(attn_output) 2025-11-03T16:32:10.9476730Z 2025-11-03T16:32:10.9476837Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9477168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9477478Z res = mod(**inputs) 2025-11-03T16:32:10.9477856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9478247Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9478627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9479015Z return self.transformer( 2025-11-03T16:32:10.9479394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9479783Z layer_outputs = layer_module( 2025-11-03T16:32:10.9480114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9480456Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9480854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9481288Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9481714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9482227Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9482741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9483126Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9483524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-11-03T16:32:10.9483923Z x = self.lin1(input) 2025-11-03T16:32:10.9484026Z 2025-11-03T16:32:10.9484133Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9484471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9484785Z res = mod(**inputs) 2025-11-03T16:32:10.9485156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9485553Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9485939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9486339Z return self.transformer( 2025-11-03T16:32:10.9486724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9487122Z layer_outputs = layer_module( 2025-11-03T16:32:10.9487467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9487828Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9488225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9488660Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9489086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9489607Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9490094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9490491Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9490892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-11-03T16:32:10.9491287Z x = self.activation(x) 2025-11-03T16:32:10.9491606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:10.9491934Z return self.act(input) 2025-11-03T16:32:10.9492047Z 2025-11-03T16:32:10.9492147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9492492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9492802Z res = mod(**inputs) 2025-11-03T16:32:10.9493167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 816, in forward 2025-11-03T16:32:10.9493560Z dlbrt_output = self.distilbert( 2025-11-03T16:32:10.9493955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 726, in forward 2025-11-03T16:32:10.9494352Z return self.transformer( 2025-11-03T16:32:10.9494721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-11-03T16:32:10.9495099Z layer_outputs = layer_module( 2025-11-03T16:32:10.9495424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:10.9495766Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:10.9496151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-11-03T16:32:10.9496598Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-11-03T16:32:10.9497013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-11-03T16:32:10.9497549Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-11-03T16:32:10.9498043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:10.9498428Z return forward_fn(*input_tensors) 2025-11-03T16:32:10.9498824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-11-03T16:32:10.9499213Z x = self.lin2(x) 2025-11-03T16:32:10.9499320Z 2025-11-03T16:32:10.9499424Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9499791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9500111Z res = mod(**inputs) 2025-11-03T16:32:10.9500488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-11-03T16:32:10.9500979Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-11-03T16:32:10.9501200Z 2025-11-03T16:32:10.9501318Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9501663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9501970Z res = mod(**inputs) 2025-11-03T16:32:10.9502330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 829, in forward 2025-11-03T16:32:10.9502830Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-11-03T16:32:10.9503064Z 2025-11-03T16:32:10.9503165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:10.9503507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:10.9503832Z res = mod(**inputs) 2025-11-03T16:32:10.9504194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 833, in forward 2025-11-03T16:32:10.9504716Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-11-03T16:32:10.9504959Z 2025-11-03T16:32:19.5661778Z Compilation time (from dynamo_timed): 12.33329277 2025-11-03T16:32:19.5663535Z pass 2025-11-03T16:32:19.5663997Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:19.5664903Z TIMING: _recursive_pre_grad_passes:0.00482 _recursive_joint_graph_passes:0.24736 _recursive_post_grad_passes:0.04175 async_compile.wait:0.74656 code_gen:8.38429 inductor_compile:9.06367 backend_compile:10.73123 gc:0.0004 entire_frame_compile:12.33329 total_wall_time:12.33329 2025-11-03T16:32:19.5666329Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:3928 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:1039 2025-11-03T16:32:19.5667125Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-11-03T16:32:21.7124991Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:32:21.7125843Z import pynvml # type: ignore[import] 2025-11-03T16:32:24.9271465Z 2025-11-03T16:32:25.8500979Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-11-03T16:32:25.8502294Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-11-03T16:32:25.8788526Z 2025-11-03T16:32:25.8794592Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:32:25.8795116Z cpu eval DistillGPT2 2025-11-03T16:32:26.2689380Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:26.4475226Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:26.6267075Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:32.6177966Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6184295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6185177Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6185627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:32:32.6186020Z causal_mask = create_causal_mask( 2025-11-03T16:32:32.6186458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:32:32.6186953Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:32:32.6187795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:32:32.6188257Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:32:32.6188716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 654, in find_packed_sequence_indices 2025-11-03T16:32:32.6189238Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-11-03T16:32:32.6189467Z 2025-11-03T16:32:32.6189561Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6189800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6190287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6190688Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6191090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6191476Z outputs = block( 2025-11-03T16:32:32.6191809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6192200Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6192641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6193036Z return func(*args, **kwargs) 2025-11-03T16:32:32.6193417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6193819Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6194392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6194810Z return func(*args, **kwargs) 2025-11-03T16:32:32.6195244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:32:32.6195782Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:32:32.6196307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6196763Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6197105Z 2025-11-03T16:32:32.6197219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6197682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6198132Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6198558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:32:32.6198975Z causal_mask = create_causal_mask( 2025-11-03T16:32:32.6199403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:32:32.6199931Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:32:32.6200462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:32:32.6200943Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:32:32.6201415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-11-03T16:32:32.6201939Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-11-03T16:32:32.6202156Z 2025-11-03T16:32:32.6202241Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6202492Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6202893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6203279Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6203647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:32:32.6204013Z causal_mask = create_causal_mask( 2025-11-03T16:32:32.6204365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:32:32.6204836Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:32:32.6205358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:32:32.6205787Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:32:32.6206233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-11-03T16:32:32.6206635Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-11-03T16:32:32.6206794Z 2025-11-03T16:32:32.6206892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6207281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6207658Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6208022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:32:32.6208376Z causal_mask = create_causal_mask( 2025-11-03T16:32:32.6208726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:32:32.6209190Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:32:32.6209666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:32:32.6210081Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:32:32.6210493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-11-03T16:32:32.6210944Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-11-03T16:32:32.6211105Z 2025-11-03T16:32:32.6211182Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6211408Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6211796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6212180Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6212551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6212914Z outputs = block( 2025-11-03T16:32:32.6213436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6213789Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6214156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6214510Z return func(*args, **kwargs) 2025-11-03T16:32:32.6214857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6215230Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6215592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6215984Z return func(*args, **kwargs) 2025-11-03T16:32:32.6216341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:32:32.6216722Z attn_output, attn_weights = attention_interface( 2025-11-03T16:32:32.6217134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:32:32.6217593Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:32.6217771Z 2025-11-03T16:32:32.6217868Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6218254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6218651Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6219004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6219351Z outputs = block( 2025-11-03T16:32:32.6219654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6219995Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6220350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6220687Z return func(*args, **kwargs) 2025-11-03T16:32:32.6221034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6221404Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6221767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6222112Z return func(*args, **kwargs) 2025-11-03T16:32:32.6222449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:32:32.6222809Z attn_output = self.c_proj(attn_output) 2025-11-03T16:32:32.6223145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6223523Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6223686Z 2025-11-03T16:32:32.6223783Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6224216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6224592Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6224954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6225304Z outputs = block( 2025-11-03T16:32:32.6225607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6225947Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6226303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6226660Z return func(*args, **kwargs) 2025-11-03T16:32:32.6227007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6227387Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6227772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:32:32.6228138Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:32:32.6228477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6228867Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6229036Z 2025-11-03T16:32:32.6229134Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6229533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6229909Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6230286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6230637Z outputs = block( 2025-11-03T16:32:32.6230950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6231317Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6231680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6232038Z return func(*args, **kwargs) 2025-11-03T16:32:32.6232384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6232779Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6233168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:32:32.6233540Z hidden_states = self.act(hidden_states) 2025-11-03T16:32:32.6233878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:32:32.6234518Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:32:32.6234761Z 2025-11-03T16:32:32.6234866Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6235284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6235670Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6236045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6236395Z outputs = block( 2025-11-03T16:32:32.6236716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6237068Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6237488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6237846Z return func(*args, **kwargs) 2025-11-03T16:32:32.6238207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6238604Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6239000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:32:32.6239381Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:32:32.6239727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6240108Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6240281Z 2025-11-03T16:32:32.6240379Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6240780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6241163Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6241531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6241891Z outputs = block( 2025-11-03T16:32:32.6242224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6242571Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6242927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6243290Z return func(*args, **kwargs) 2025-11-03T16:32:32.6243645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6244027Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6244400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6244778Z return func(*args, **kwargs) 2025-11-03T16:32:32.6245140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:32:32.6245604Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:32:32.6246035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6246404Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6246566Z 2025-11-03T16:32:32.6246643Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6246880Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6247273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6247644Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6248010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6248352Z outputs = block( 2025-11-03T16:32:32.6248653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6248996Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6249348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6249691Z return func(*args, **kwargs) 2025-11-03T16:32:32.6250038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6250440Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6250809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6251164Z return func(*args, **kwargs) 2025-11-03T16:32:32.6251507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:32:32.6251897Z attn_output, attn_weights = attention_interface( 2025-11-03T16:32:32.6252322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:32:32.6252779Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:32.6252956Z 2025-11-03T16:32:32.6253062Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6253453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6253832Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6254202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6254554Z outputs = block( 2025-11-03T16:32:32.6254855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6255213Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6255565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6255911Z return func(*args, **kwargs) 2025-11-03T16:32:32.6256253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6256616Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6256978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6257325Z return func(*args, **kwargs) 2025-11-03T16:32:32.6257669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:32:32.6258050Z attn_output = self.c_proj(attn_output) 2025-11-03T16:32:32.6258376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6258750Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6258918Z 2025-11-03T16:32:32.6259015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6259402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6259778Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6260139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6260500Z outputs = block( 2025-11-03T16:32:32.6260805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6261146Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6261491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6261843Z return func(*args, **kwargs) 2025-11-03T16:32:32.6262188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6262572Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6262956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:32:32.6263309Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:32:32.6263677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6264049Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6264208Z 2025-11-03T16:32:32.6264314Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6264705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6265070Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6265433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6265779Z outputs = block( 2025-11-03T16:32:32.6266082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6266412Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6266763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6267116Z return func(*args, **kwargs) 2025-11-03T16:32:32.6267459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6267838Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6268235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:32:32.6268603Z hidden_states = self.act(hidden_states) 2025-11-03T16:32:32.6268934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:32:32.6269366Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:32:32.6269581Z 2025-11-03T16:32:32.6269688Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6270071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6270457Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6270820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6271169Z outputs = block( 2025-11-03T16:32:32.6271474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6271814Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6272208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6272570Z return func(*args, **kwargs) 2025-11-03T16:32:32.6272927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6273316Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6273705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:32:32.6274178Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:32:32.6274524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6274917Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6275084Z 2025-11-03T16:32:32.6275185Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6275602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6276001Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6276426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6276829Z outputs = block( 2025-11-03T16:32:32.6277142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6277500Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6277866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6278231Z return func(*args, **kwargs) 2025-11-03T16:32:32.6278581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6278961Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6279335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6279694Z return func(*args, **kwargs) 2025-11-03T16:32:32.6280051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:32:32.6280515Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:32:32.6280963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6281348Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6281534Z 2025-11-03T16:32:32.6281621Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6281850Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6282259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6282654Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6283053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6283419Z outputs = block( 2025-11-03T16:32:32.6283732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6284133Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6284498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6284857Z return func(*args, **kwargs) 2025-11-03T16:32:32.6285210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6285581Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6285954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6286312Z return func(*args, **kwargs) 2025-11-03T16:32:32.6286664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:32:32.6287051Z attn_output, attn_weights = attention_interface( 2025-11-03T16:32:32.6287469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:32:32.6287930Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:32.6288113Z 2025-11-03T16:32:32.6288210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6288613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6288995Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6289361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6289718Z outputs = block( 2025-11-03T16:32:32.6290064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6290414Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6290781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6291140Z return func(*args, **kwargs) 2025-11-03T16:32:32.6291498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6291881Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6292264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6292607Z return func(*args, **kwargs) 2025-11-03T16:32:32.6292956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:32:32.6293327Z attn_output = self.c_proj(attn_output) 2025-11-03T16:32:32.6293665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6294039Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6294200Z 2025-11-03T16:32:32.6294297Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6294686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6295106Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6295463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6295806Z outputs = block( 2025-11-03T16:32:32.6296112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6296452Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6296803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6297176Z return func(*args, **kwargs) 2025-11-03T16:32:32.6297511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6297895Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6298277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:32:32.6298636Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:32:32.6298968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6299331Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6299495Z 2025-11-03T16:32:32.6299594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6299979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6300349Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6300702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6301050Z outputs = block( 2025-11-03T16:32:32.6301350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6301687Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6302034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6302369Z return func(*args, **kwargs) 2025-11-03T16:32:32.6302743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6303130Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6303508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:32:32.6303868Z hidden_states = self.act(hidden_states) 2025-11-03T16:32:32.6304190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:32:32.6304616Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:32:32.6304840Z 2025-11-03T16:32:32.6304937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6305323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6305688Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6306041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6306399Z outputs = block( 2025-11-03T16:32:32.6306704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6307040Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6307385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6307755Z return func(*args, **kwargs) 2025-11-03T16:32:32.6308104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6308494Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6308876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:32:32.6309242Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:32:32.6309590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6309997Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6310162Z 2025-11-03T16:32:32.6310270Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6310671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6311044Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6311416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6311777Z outputs = block( 2025-11-03T16:32:32.6312088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6312440Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6312798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6313161Z return func(*args, **kwargs) 2025-11-03T16:32:32.6313695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:32:32.6314143Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:32:32.6314303Z 2025-11-03T16:32:32.6314402Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6314805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6315190Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6315573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6316020Z outputs = block( 2025-11-03T16:32:32.6316328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6316681Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6317085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6317450Z return func(*args, **kwargs) 2025-11-03T16:32:32.6317813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6318188Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6318569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6318933Z return func(*args, **kwargs) 2025-11-03T16:32:32.6319294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:32:32.6319775Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:32:32.6320223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6320607Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6320772Z 2025-11-03T16:32:32.6320885Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6321120Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6321548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6321961Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6322365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6322755Z outputs = block( 2025-11-03T16:32:32.6323096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6323478Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6323900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6324287Z return func(*args, **kwargs) 2025-11-03T16:32:32.6324661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6325101Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6325465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6325812Z return func(*args, **kwargs) 2025-11-03T16:32:32.6326153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:32:32.6326533Z attn_output, attn_weights = attention_interface( 2025-11-03T16:32:32.6326941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:32:32.6327404Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:32.6327579Z 2025-11-03T16:32:32.6327673Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6328062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6328433Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6328790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6329136Z outputs = block( 2025-11-03T16:32:32.6329471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6329818Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6330172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6330516Z return func(*args, **kwargs) 2025-11-03T16:32:32.6330861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6331231Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6331596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6331935Z return func(*args, **kwargs) 2025-11-03T16:32:32.6332282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:32:32.6332646Z attn_output = self.c_proj(attn_output) 2025-11-03T16:32:32.6332991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6333367Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6333532Z 2025-11-03T16:32:32.6333629Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6334018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6334415Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6334775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6335120Z outputs = block( 2025-11-03T16:32:32.6335414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6335753Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6336110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6336456Z return func(*args, **kwargs) 2025-11-03T16:32:32.6336812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6337206Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6337589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:32:32.6337964Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:32:32.6338303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6338676Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6338845Z 2025-11-03T16:32:32.6338942Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6339338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6339719Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6340097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6340438Z outputs = block( 2025-11-03T16:32:32.6340741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6341085Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6341447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6341796Z return func(*args, **kwargs) 2025-11-03T16:32:32.6342151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6342584Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6342978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:32:32.6343350Z hidden_states = self.act(hidden_states) 2025-11-03T16:32:32.6343680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:32:32.6344113Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:32:32.6344340Z 2025-11-03T16:32:32.6344442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6344839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6345219Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6345588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6345944Z outputs = block( 2025-11-03T16:32:32.6346254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6346602Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6346964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6347338Z return func(*args, **kwargs) 2025-11-03T16:32:32.6347695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6348090Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6348483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:32:32.6348855Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:32:32.6349203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6349586Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6349770Z 2025-11-03T16:32:32.6349877Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6350274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6350641Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6351011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6351366Z outputs = block( 2025-11-03T16:32:32.6351677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6352021Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6352376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6352730Z return func(*args, **kwargs) 2025-11-03T16:32:32.6353082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6353456Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6353820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6354252Z return func(*args, **kwargs) 2025-11-03T16:32:32.6354618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:32:32.6355113Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:32:32.6355570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6355997Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6356174Z 2025-11-03T16:32:32.6356254Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6356488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6356890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6357282Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6357653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6358038Z outputs = block( 2025-11-03T16:32:32.6358371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6358711Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6359065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6359407Z return func(*args, **kwargs) 2025-11-03T16:32:32.6359749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6360118Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6360479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6360833Z return func(*args, **kwargs) 2025-11-03T16:32:32.6361177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:32:32.6361561Z attn_output, attn_weights = attention_interface( 2025-11-03T16:32:32.6361988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:32:32.6362449Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:32.6362621Z 2025-11-03T16:32:32.6362722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6363137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6363516Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6363894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6364249Z outputs = block( 2025-11-03T16:32:32.6364545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6364885Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6365234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6365580Z return func(*args, **kwargs) 2025-11-03T16:32:32.6365918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6366285Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6366645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6366989Z return func(*args, **kwargs) 2025-11-03T16:32:32.6367330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:32:32.6367693Z attn_output = self.c_proj(attn_output) 2025-11-03T16:32:32.6368032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6368422Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6368589Z 2025-11-03T16:32:32.6368729Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6369130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6369504Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6369877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6370233Z outputs = block( 2025-11-03T16:32:32.6370548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6370895Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6371251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6371605Z return func(*args, **kwargs) 2025-11-03T16:32:32.6371959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6372356Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6372741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:32:32.6373117Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:32:32.6373460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6373862Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6374023Z 2025-11-03T16:32:32.6374130Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6374519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6374900Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6375273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6375631Z outputs = block( 2025-11-03T16:32:32.6375945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6376306Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6376670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6377030Z return func(*args, **kwargs) 2025-11-03T16:32:32.6377385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6377770Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6378164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:32:32.6378535Z hidden_states = self.act(hidden_states) 2025-11-03T16:32:32.6378877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:32:32.6379310Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:32:32.6379529Z 2025-11-03T16:32:32.6379629Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6380026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6380404Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6380775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6381134Z outputs = block( 2025-11-03T16:32:32.6381440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6381785Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6382196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6382565Z return func(*args, **kwargs) 2025-11-03T16:32:32.6382945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6383355Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6383760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:32:32.6384149Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:32:32.6384508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6384892Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6385070Z 2025-11-03T16:32:32.6385172Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6385586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6386000Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6386401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6386781Z outputs = block( 2025-11-03T16:32:32.6387141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6387522Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6387959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6388314Z return func(*args, **kwargs) 2025-11-03T16:32:32.6388656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:32:32.6389049Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:32:32.6389207Z 2025-11-03T16:32:32.6389306Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6389712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6390079Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6390434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6390779Z outputs = block( 2025-11-03T16:32:32.6391083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6391418Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6391763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6392113Z return func(*args, **kwargs) 2025-11-03T16:32:32.6392459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6392833Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6393199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6393550Z return func(*args, **kwargs) 2025-11-03T16:32:32.6393893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:32:32.6394452Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:32:32.6394906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6395314Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6395520Z 2025-11-03T16:32:32.6395603Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6395812Z cudagraph partition due to non gpu ops 2025-11-03T16:32:32.6396042Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6396433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6396797Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6397164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6397511Z outputs = block( 2025-11-03T16:32:32.6397818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6398157Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6398508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6398856Z return func(*args, **kwargs) 2025-11-03T16:32:32.6399200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6399569Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6399925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6400291Z return func(*args, **kwargs) 2025-11-03T16:32:32.6400635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:32:32.6401015Z attn_output, attn_weights = attention_interface( 2025-11-03T16:32:32.6401432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:32:32.6401876Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:32:32.6402053Z 2025-11-03T16:32:32.6402149Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6402536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6402922Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6403281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6403622Z outputs = block( 2025-11-03T16:32:32.6403925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6404265Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6404616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6404962Z return func(*args, **kwargs) 2025-11-03T16:32:32.6405301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:32:32.6405671Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:32:32.6406035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6406383Z return func(*args, **kwargs) 2025-11-03T16:32:32.6406720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:32:32.6407083Z attn_output = self.c_proj(attn_output) 2025-11-03T16:32:32.6407418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6407786Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6407949Z 2025-11-03T16:32:32.6408053Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6408466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6408844Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6409211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6409564Z outputs = block( 2025-11-03T16:32:32.6409868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6410203Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6410562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6410914Z return func(*args, **kwargs) 2025-11-03T16:32:32.6411263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6411649Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6412033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:32:32.6412404Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:32:32.6412740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6413127Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6413427Z 2025-11-03T16:32:32.6413531Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6413924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6414294Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6414658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6415013Z outputs = block( 2025-11-03T16:32:32.6415318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6415703Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6416059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6416406Z return func(*args, **kwargs) 2025-11-03T16:32:32.6416742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6417126Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6417503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:32:32.6417862Z hidden_states = self.act(hidden_states) 2025-11-03T16:32:32.6418196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:32:32.6418612Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:32:32.6418840Z 2025-11-03T16:32:32.6418939Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6419337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1070, in forward 2025-11-03T16:32:32.6419711Z transformer_outputs = self.transformer( 2025-11-03T16:32:32.6420071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:32:32.6420410Z outputs = block( 2025-11-03T16:32:32.6420712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:32.6421052Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:32.6421459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:32.6421818Z return func(*args, **kwargs) 2025-11-03T16:32:32.6422160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:32:32.6422544Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:32:32.6422924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:32:32.6423295Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:32:32.6423624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:32:32.6423993Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:32:32.6424158Z 2025-11-03T16:32:32.6424253Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:32.6424644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1094, in forward 2025-11-03T16:32:32.6425044Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-11-03T16:32:32.6425198Z 2025-11-03T16:32:41.1009469Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:41.1010685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-11-03T16:32:41.1011607Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-11-03T16:32:41.1012164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-11-03T16:32:41.1012716Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-11-03T16:32:41.1013015Z 2025-11-03T16:32:42.0931552Z Compilation time (from dynamo_timed): 14.814868625 2025-11-03T16:32:42.1075816Z pass 2025-11-03T16:32:42.1080381Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:42.1082036Z TIMING: gc:0.00384 entire_frame_compile:14.81487 _recursive_pre_grad_passes:0.00667 _recursive_joint_graph_passes:0.47558 _recursive_post_grad_passes:0.04434 async_compile.wait:1.33176 code_gen:8.91674 inductor_compile:9.60217 backend_compile:11.46253 total_wall_time:14.81487 2025-11-03T16:32:42.1082956Z STATS: call_* op count: 311 | FakeTensorMode.__torch_dispatch__:4742 | FakeTensor.__torch_dispatch__:2291 | ProxyTorchDispatchMode.__torch_dispatch__:903 2025-11-03T16:32:42.1083452Z Dynamo produced 2 graphs covering 311 ops with 2 graph breaks (1 unique) 2025-11-03T16:32:44.2525827Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:32:44.2526905Z import pynvml # type: ignore[import] 2025-11-03T16:32:47.3968650Z 2025-11-03T16:32:47.3981153Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-11-03T16:32:47.3981927Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-11-03T16:32:47.5992408Z 2025-11-03T16:32:47.5992580Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:32:47.6005593Z cpu eval ElectraForCausalLM 2025-11-03T16:32:47.7801755Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:47.8620083Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:47.9397697Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:32:55.4513564Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4514783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4515115Z res = mod(**inputs) 2025-11-03T16:32:55.4515530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4515940Z outputs = self.electra( 2025-11-03T16:32:55.4516316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 787, in forward 2025-11-03T16:32:55.4516724Z hidden_states = self.embeddings_project(hidden_states) 2025-11-03T16:32:55.4516890Z 2025-11-03T16:32:55.4516995Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4517341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4517645Z res = mod(**inputs) 2025-11-03T16:32:55.4518010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4518386Z outputs = self.electra( 2025-11-03T16:32:55.4518739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4519116Z hidden_states = self.encoder( 2025-11-03T16:32:55.4519486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4519936Z layer_outputs = layer_module( 2025-11-03T16:32:55.4520269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4520627Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4521004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4521367Z return func(*args, **kwargs) 2025-11-03T16:32:55.4521740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4522172Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4522564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4523043Z return func(*args, **kwargs) 2025-11-03T16:32:55.4523490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4523860Z self_outputs = self.self( 2025-11-03T16:32:55.4524202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4524547Z return func(*args, **kwargs) 2025-11-03T16:32:55.4524918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4525299Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4525436Z 2025-11-03T16:32:55.4525548Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4525975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4526428Z res = mod(**inputs) 2025-11-03T16:32:55.4526801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4527179Z outputs = self.electra( 2025-11-03T16:32:55.4527541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4527913Z hidden_states = self.encoder( 2025-11-03T16:32:55.4528360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4528739Z layer_outputs = layer_module( 2025-11-03T16:32:55.4529075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4529461Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4529832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4530197Z return func(*args, **kwargs) 2025-11-03T16:32:55.4530574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4530945Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4531305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4531655Z return func(*args, **kwargs) 2025-11-03T16:32:55.4532009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4532414Z self_outputs = self.self( 2025-11-03T16:32:55.4532855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4533209Z return func(*args, **kwargs) 2025-11-03T16:32:55.4533592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4534146Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4534272Z 2025-11-03T16:32:55.4534378Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4534711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4535011Z res = mod(**inputs) 2025-11-03T16:32:55.4535361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4535722Z outputs = self.electra( 2025-11-03T16:32:55.4536063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4536457Z hidden_states = self.encoder( 2025-11-03T16:32:55.4536812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4537181Z layer_outputs = layer_module( 2025-11-03T16:32:55.4537504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4537842Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4538200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4538550Z return func(*args, **kwargs) 2025-11-03T16:32:55.4538938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4539439Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4539798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4540143Z return func(*args, **kwargs) 2025-11-03T16:32:55.4540500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4540869Z self_outputs = self.self( 2025-11-03T16:32:55.4541227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4541583Z return func(*args, **kwargs) 2025-11-03T16:32:55.4541945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4542433Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4542626Z 2025-11-03T16:32:55.4542751Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4542986Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4543217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4543679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4543991Z res = mod(**inputs) 2025-11-03T16:32:55.4544345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4544727Z outputs = self.electra( 2025-11-03T16:32:55.4545084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4545465Z hidden_states = self.encoder( 2025-11-03T16:32:55.4545843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4546206Z layer_outputs = layer_module( 2025-11-03T16:32:55.4546533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4546875Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4547247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4547799Z return func(*args, **kwargs) 2025-11-03T16:32:55.4548167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4548562Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4548936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4549295Z return func(*args, **kwargs) 2025-11-03T16:32:55.4549669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4550113Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4550578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4550965Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4551102Z 2025-11-03T16:32:55.4551249Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4551706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4552021Z res = mod(**inputs) 2025-11-03T16:32:55.4552386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4552771Z outputs = self.electra( 2025-11-03T16:32:55.4553140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4553528Z hidden_states = self.encoder( 2025-11-03T16:32:55.4553899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4554365Z layer_outputs = layer_module( 2025-11-03T16:32:55.4554701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4555047Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4555415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4555777Z return func(*args, **kwargs) 2025-11-03T16:32:55.4556145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4556665Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4557121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4557501Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4557910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4558370Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4558813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4559202Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4559345Z 2025-11-03T16:32:55.4559701Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4560200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4560513Z res = mod(**inputs) 2025-11-03T16:32:55.4560867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4561237Z outputs = self.electra( 2025-11-03T16:32:55.4561595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4561971Z hidden_states = self.encoder( 2025-11-03T16:32:55.4562360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4562723Z layer_outputs = layer_module( 2025-11-03T16:32:55.4563056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4563403Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4563766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4564120Z return func(*args, **kwargs) 2025-11-03T16:32:55.4564477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4564916Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4565311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4565700Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4566121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4566580Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4567163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4567670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4568075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4568419Z return self.act(input) 2025-11-03T16:32:55.4568532Z 2025-11-03T16:32:55.4568636Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4569004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4569326Z res = mod(**inputs) 2025-11-03T16:32:55.4569692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4570072Z outputs = self.electra( 2025-11-03T16:32:55.4570440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4570826Z hidden_states = self.encoder( 2025-11-03T16:32:55.4571416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4571801Z layer_outputs = layer_module( 2025-11-03T16:32:55.4572142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4572504Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4572879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4573248Z return func(*args, **kwargs) 2025-11-03T16:32:55.4573616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4574016Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4574410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4574798Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4575216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4575685Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4576133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4576555Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4576689Z 2025-11-03T16:32:55.4576800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4577155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4577471Z res = mod(**inputs) 2025-11-03T16:32:55.4577843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4578236Z outputs = self.electra( 2025-11-03T16:32:55.4578610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4579016Z hidden_states = self.encoder( 2025-11-03T16:32:55.4579391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4579776Z layer_outputs = layer_module( 2025-11-03T16:32:55.4580117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4580476Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4580854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4581214Z return func(*args, **kwargs) 2025-11-03T16:32:55.4581597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4581992Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4582369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4582725Z return func(*args, **kwargs) 2025-11-03T16:32:55.4583104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4583506Z self_outputs = self.self( 2025-11-03T16:32:55.4583873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4584242Z return func(*args, **kwargs) 2025-11-03T16:32:55.4584613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4585046Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4585179Z 2025-11-03T16:32:55.4585284Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4585629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4585932Z res = mod(**inputs) 2025-11-03T16:32:55.4586293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4586674Z outputs = self.electra( 2025-11-03T16:32:55.4587033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4587411Z hidden_states = self.encoder( 2025-11-03T16:32:55.4587774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4588149Z layer_outputs = layer_module( 2025-11-03T16:32:55.4588485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4588837Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4589201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4589574Z return func(*args, **kwargs) 2025-11-03T16:32:55.4589949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4590419Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4590797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4591165Z return func(*args, **kwargs) 2025-11-03T16:32:55.4591552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4591941Z self_outputs = self.self( 2025-11-03T16:32:55.4592294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4592694Z return func(*args, **kwargs) 2025-11-03T16:32:55.4593059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4593449Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4593591Z 2025-11-03T16:32:55.4593693Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4594135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4594461Z res = mod(**inputs) 2025-11-03T16:32:55.4594849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4595254Z outputs = self.electra( 2025-11-03T16:32:55.4595628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4596013Z hidden_states = self.encoder( 2025-11-03T16:32:55.4596386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4596774Z layer_outputs = layer_module( 2025-11-03T16:32:55.4597112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4597472Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4597847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4598203Z return func(*args, **kwargs) 2025-11-03T16:32:55.4598580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4599015Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4599380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4599738Z return func(*args, **kwargs) 2025-11-03T16:32:55.4600090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4600453Z self_outputs = self.self( 2025-11-03T16:32:55.4600787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4601128Z return func(*args, **kwargs) 2025-11-03T16:32:55.4601472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4601844Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4601976Z 2025-11-03T16:32:55.4602055Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4602261Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4602473Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4602824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4603130Z res = mod(**inputs) 2025-11-03T16:32:55.4603486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4603872Z outputs = self.electra( 2025-11-03T16:32:55.4604215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4604578Z hidden_states = self.encoder( 2025-11-03T16:32:55.4604939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4605301Z layer_outputs = layer_module( 2025-11-03T16:32:55.4605617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4605960Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4606333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4606679Z return func(*args, **kwargs) 2025-11-03T16:32:55.4607032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4607405Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4607757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4608101Z return func(*args, **kwargs) 2025-11-03T16:32:55.4608455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4608871Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4609280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4609658Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4609795Z 2025-11-03T16:32:55.4609892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4610224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4610525Z res = mod(**inputs) 2025-11-03T16:32:55.4610866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4611234Z outputs = self.electra( 2025-11-03T16:32:55.4611591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4611999Z hidden_states = self.encoder( 2025-11-03T16:32:55.4612352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4612720Z layer_outputs = layer_module( 2025-11-03T16:32:55.4613047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4613593Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4613957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4614303Z return func(*args, **kwargs) 2025-11-03T16:32:55.4614660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4615039Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4615419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4615796Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4616190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4616643Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4617055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4617493Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4617623Z 2025-11-03T16:32:55.4617719Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4618056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4618362Z res = mod(**inputs) 2025-11-03T16:32:55.4618721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4619090Z outputs = self.electra( 2025-11-03T16:32:55.4619436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4619838Z hidden_states = self.encoder( 2025-11-03T16:32:55.4620203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4620568Z layer_outputs = layer_module( 2025-11-03T16:32:55.4620892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4621224Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4621581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4621926Z return func(*args, **kwargs) 2025-11-03T16:32:55.4622285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4622659Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4623034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4623400Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4623796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4624238Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4624643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4625045Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4625457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4625780Z return self.act(input) 2025-11-03T16:32:55.4625885Z 2025-11-03T16:32:55.4625992Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4626318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4626621Z res = mod(**inputs) 2025-11-03T16:32:55.4626972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4627336Z outputs = self.electra( 2025-11-03T16:32:55.4627677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4628043Z hidden_states = self.encoder( 2025-11-03T16:32:55.4628398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4628761Z layer_outputs = layer_module( 2025-11-03T16:32:55.4629084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4629416Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4629772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4630138Z return func(*args, **kwargs) 2025-11-03T16:32:55.4630492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4630870Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4631239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4631606Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4632010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4632465Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4632890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4633273Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4633411Z 2025-11-03T16:32:55.4633510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4633853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4634234Z res = mod(**inputs) 2025-11-03T16:32:55.4634598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4634987Z outputs = self.electra( 2025-11-03T16:32:55.4635366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4635749Z hidden_states = self.encoder( 2025-11-03T16:32:55.4636175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4636545Z layer_outputs = layer_module( 2025-11-03T16:32:55.4636883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4637239Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4637610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4637964Z return func(*args, **kwargs) 2025-11-03T16:32:55.4638333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4638755Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4639136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4639501Z return func(*args, **kwargs) 2025-11-03T16:32:55.4639869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4640254Z self_outputs = self.self( 2025-11-03T16:32:55.4640610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4640974Z return func(*args, **kwargs) 2025-11-03T16:32:55.4641346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4641734Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4641876Z 2025-11-03T16:32:55.4641978Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4642332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4642653Z res = mod(**inputs) 2025-11-03T16:32:55.4643015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4643399Z outputs = self.electra( 2025-11-03T16:32:55.4643766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4644170Z hidden_states = self.encoder( 2025-11-03T16:32:55.4644538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4644907Z layer_outputs = layer_module( 2025-11-03T16:32:55.4645235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4645584Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4645946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4646325Z return func(*args, **kwargs) 2025-11-03T16:32:55.4646684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4647073Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4647432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4647785Z return func(*args, **kwargs) 2025-11-03T16:32:55.4648174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4648537Z self_outputs = self.self( 2025-11-03T16:32:55.4648879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4649228Z return func(*args, **kwargs) 2025-11-03T16:32:55.4649594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4649970Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4650105Z 2025-11-03T16:32:55.4650205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4650549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4650861Z res = mod(**inputs) 2025-11-03T16:32:55.4651211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4651589Z outputs = self.electra( 2025-11-03T16:32:55.4651947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4652355Z hidden_states = self.encoder( 2025-11-03T16:32:55.4652724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4653093Z layer_outputs = layer_module( 2025-11-03T16:32:55.4653426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4653778Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4654139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4654494Z return func(*args, **kwargs) 2025-11-03T16:32:55.4654848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4655232Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4655603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4655956Z return func(*args, **kwargs) 2025-11-03T16:32:55.4656325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4656701Z self_outputs = self.self( 2025-11-03T16:32:55.4657046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4657426Z return func(*args, **kwargs) 2025-11-03T16:32:55.4657800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4658171Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4658307Z 2025-11-03T16:32:55.4658388Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4658597Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4658830Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4659172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4659494Z res = mod(**inputs) 2025-11-03T16:32:55.4659856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4660235Z outputs = self.electra( 2025-11-03T16:32:55.4660603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4660970Z hidden_states = self.encoder( 2025-11-03T16:32:55.4661341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4661716Z layer_outputs = layer_module( 2025-11-03T16:32:55.4662045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4662394Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4662747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4663102Z return func(*args, **kwargs) 2025-11-03T16:32:55.4663462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4663848Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4664207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4664564Z return func(*args, **kwargs) 2025-11-03T16:32:55.4664930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4665361Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4665817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4666200Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4666342Z 2025-11-03T16:32:55.4666441Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4666787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4667099Z res = mod(**inputs) 2025-11-03T16:32:55.4667462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4667835Z outputs = self.electra( 2025-11-03T16:32:55.4668196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4668573Z hidden_states = self.encoder( 2025-11-03T16:32:55.4668944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4669313Z layer_outputs = layer_module( 2025-11-03T16:32:55.4669645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4670000Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4670366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4670772Z return func(*args, **kwargs) 2025-11-03T16:32:55.4671127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4671517Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4671909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4672292Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4672709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4673184Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4673625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4674113Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4674255Z 2025-11-03T16:32:55.4674369Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4674735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4675053Z res = mod(**inputs) 2025-11-03T16:32:55.4675440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4675827Z outputs = self.electra( 2025-11-03T16:32:55.4676205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4676586Z hidden_states = self.encoder( 2025-11-03T16:32:55.4676968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4677356Z layer_outputs = layer_module( 2025-11-03T16:32:55.4677701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4678064Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4678429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4678799Z return func(*args, **kwargs) 2025-11-03T16:32:55.4679208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4679614Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4680009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4680401Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4680837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4681319Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4681759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4682182Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4682567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4682917Z return self.act(input) 2025-11-03T16:32:55.4683032Z 2025-11-03T16:32:55.4683145Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4683505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4683825Z res = mod(**inputs) 2025-11-03T16:32:55.4684198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4684604Z outputs = self.electra( 2025-11-03T16:32:55.4684973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4685361Z hidden_states = self.encoder( 2025-11-03T16:32:55.4685733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4686116Z layer_outputs = layer_module( 2025-11-03T16:32:55.4686461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4686819Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4687211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4687593Z return func(*args, **kwargs) 2025-11-03T16:32:55.4687984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4688393Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4688798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4689185Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4689610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4690100Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4690555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4690961Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4691100Z 2025-11-03T16:32:55.4691204Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4691570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4691899Z res = mod(**inputs) 2025-11-03T16:32:55.4692273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4692661Z outputs = self.electra( 2025-11-03T16:32:55.4693036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4693481Z hidden_states = self.encoder( 2025-11-03T16:32:55.4693842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4694209Z layer_outputs = layer_module( 2025-11-03T16:32:55.4694526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4694866Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4695222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4695572Z return func(*args, **kwargs) 2025-11-03T16:32:55.4695927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4696293Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4696651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4696998Z return func(*args, **kwargs) 2025-11-03T16:32:55.4697354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4697713Z self_outputs = self.self( 2025-11-03T16:32:55.4698048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4698408Z return func(*args, **kwargs) 2025-11-03T16:32:55.4698762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4699135Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4699262Z 2025-11-03T16:32:55.4699360Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4699692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4699992Z res = mod(**inputs) 2025-11-03T16:32:55.4700337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4700720Z outputs = self.electra( 2025-11-03T16:32:55.4701064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4701429Z hidden_states = self.encoder( 2025-11-03T16:32:55.4701789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4702150Z layer_outputs = layer_module( 2025-11-03T16:32:55.4702466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4702803Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4703161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4703506Z return func(*args, **kwargs) 2025-11-03T16:32:55.4703858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4704226Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4704583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4704929Z return func(*args, **kwargs) 2025-11-03T16:32:55.4705282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4705642Z self_outputs = self.self( 2025-11-03T16:32:55.4705972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4706314Z return func(*args, **kwargs) 2025-11-03T16:32:55.4706696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4707067Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4707189Z 2025-11-03T16:32:55.4707286Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4707620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4707931Z res = mod(**inputs) 2025-11-03T16:32:55.4708276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4708639Z outputs = self.electra( 2025-11-03T16:32:55.4708978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4709343Z hidden_states = self.encoder( 2025-11-03T16:32:55.4709706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4710081Z layer_outputs = layer_module( 2025-11-03T16:32:55.4710408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4710754Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4711116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4711485Z return func(*args, **kwargs) 2025-11-03T16:32:55.4711848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4712223Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4712595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4712953Z return func(*args, **kwargs) 2025-11-03T16:32:55.4713448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4713880Z self_outputs = self.self( 2025-11-03T16:32:55.4714283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4714656Z return func(*args, **kwargs) 2025-11-03T16:32:55.4715049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4715465Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4715597Z 2025-11-03T16:32:55.4715677Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4715886Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4716113Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4716468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4716791Z res = mod(**inputs) 2025-11-03T16:32:55.4717148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4717532Z outputs = self.electra( 2025-11-03T16:32:55.4717900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4718280Z hidden_states = self.encoder( 2025-11-03T16:32:55.4718649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4719063Z layer_outputs = layer_module( 2025-11-03T16:32:55.4719395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4719746Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4720167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4720519Z return func(*args, **kwargs) 2025-11-03T16:32:55.4720893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4721308Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4721682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4722044Z return func(*args, **kwargs) 2025-11-03T16:32:55.4722410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4722855Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4723293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4723696Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4723830Z 2025-11-03T16:32:55.4723928Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4724277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4724590Z res = mod(**inputs) 2025-11-03T16:32:55.4724954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4725411Z outputs = self.electra( 2025-11-03T16:32:55.4725763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4726138Z hidden_states = self.encoder( 2025-11-03T16:32:55.4726503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4726879Z layer_outputs = layer_module( 2025-11-03T16:32:55.4727211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4727566Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4727928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4728283Z return func(*args, **kwargs) 2025-11-03T16:32:55.4728692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4729062Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4729435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4729798Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4730197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4730636Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4731039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4731411Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4731544Z 2025-11-03T16:32:55.4731640Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4731971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4732265Z res = mod(**inputs) 2025-11-03T16:32:55.4732602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4732967Z outputs = self.electra( 2025-11-03T16:32:55.4733354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4733720Z hidden_states = self.encoder( 2025-11-03T16:32:55.4734070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4734439Z layer_outputs = layer_module( 2025-11-03T16:32:55.4734768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4735110Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4735468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4735812Z return func(*args, **kwargs) 2025-11-03T16:32:55.4736170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4736549Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4736927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4737295Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4737691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4738134Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4738576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4738990Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4739358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4739682Z return self.act(input) 2025-11-03T16:32:55.4739794Z 2025-11-03T16:32:55.4739893Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4740241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4740553Z res = mod(**inputs) 2025-11-03T16:32:55.4740918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4741294Z outputs = self.electra( 2025-11-03T16:32:55.4741654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4742032Z hidden_states = self.encoder( 2025-11-03T16:32:55.4742402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4742768Z layer_outputs = layer_module( 2025-11-03T16:32:55.4743102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4743459Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4743825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4744175Z return func(*args, **kwargs) 2025-11-03T16:32:55.4744541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4744932Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4745313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4745686Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4746084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4746543Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4747015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4747408Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4747544Z 2025-11-03T16:32:55.4747651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4747997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4748316Z res = mod(**inputs) 2025-11-03T16:32:55.4748680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4749070Z outputs = self.electra( 2025-11-03T16:32:55.4749441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4749819Z hidden_states = self.encoder( 2025-11-03T16:32:55.4750201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4750589Z layer_outputs = layer_module( 2025-11-03T16:32:55.4750928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4751284Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4751661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4752048Z return func(*args, **kwargs) 2025-11-03T16:32:55.4752423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4752816Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4753186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4753554Z return func(*args, **kwargs) 2025-11-03T16:32:55.4754001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4754433Z self_outputs = self.self( 2025-11-03T16:32:55.4754794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4755174Z return func(*args, **kwargs) 2025-11-03T16:32:55.4755566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4755963Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4756097Z 2025-11-03T16:32:55.4756207Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4756556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4756880Z res = mod(**inputs) 2025-11-03T16:32:55.4757252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4757641Z outputs = self.electra( 2025-11-03T16:32:55.4758009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4758384Z hidden_states = self.encoder( 2025-11-03T16:32:55.4758764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4759148Z layer_outputs = layer_module( 2025-11-03T16:32:55.4759487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4759837Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4760210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4760574Z return func(*args, **kwargs) 2025-11-03T16:32:55.4760978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4761376Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4761746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4762112Z return func(*args, **kwargs) 2025-11-03T16:32:55.4762487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4762869Z self_outputs = self.self( 2025-11-03T16:32:55.4763223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4763577Z return func(*args, **kwargs) 2025-11-03T16:32:55.4763948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4764341Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4764473Z 2025-11-03T16:32:55.4764579Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4764922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4765236Z res = mod(**inputs) 2025-11-03T16:32:55.4765599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4766003Z outputs = self.electra( 2025-11-03T16:32:55.4766375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4766757Z hidden_states = self.encoder( 2025-11-03T16:32:55.4767138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4767531Z layer_outputs = layer_module( 2025-11-03T16:32:55.4767879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4768254Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4768618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4768980Z return func(*args, **kwargs) 2025-11-03T16:32:55.4769353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4769748Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4770113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4770476Z return func(*args, **kwargs) 2025-11-03T16:32:55.4770854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4771235Z self_outputs = self.self( 2025-11-03T16:32:55.4771589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4771947Z return func(*args, **kwargs) 2025-11-03T16:32:55.4772316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4772713Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4772843Z 2025-11-03T16:32:55.4772931Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4773136Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4773365Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4773715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4774031Z res = mod(**inputs) 2025-11-03T16:32:55.4774428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4774810Z outputs = self.electra( 2025-11-03T16:32:55.4775182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4775568Z hidden_states = self.encoder( 2025-11-03T16:32:55.4775944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4776326Z layer_outputs = layer_module( 2025-11-03T16:32:55.4776661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4777014Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4777386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4777754Z return func(*args, **kwargs) 2025-11-03T16:32:55.4778119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4778520Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4778895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4779258Z return func(*args, **kwargs) 2025-11-03T16:32:55.4779648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4780082Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4780519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4780913Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4781046Z 2025-11-03T16:32:55.4781157Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4781505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4781835Z res = mod(**inputs) 2025-11-03T16:32:55.4782198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4782584Z outputs = self.electra( 2025-11-03T16:32:55.4782975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4783354Z hidden_states = self.encoder( 2025-11-03T16:32:55.4783728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4784110Z layer_outputs = layer_module( 2025-11-03T16:32:55.4784451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4784805Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4785168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4785537Z return func(*args, **kwargs) 2025-11-03T16:32:55.4785914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4786315Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4786699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4787085Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4787503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4788004Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4788428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4788807Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4788942Z 2025-11-03T16:32:55.4789045Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4789384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4789699Z res = mod(**inputs) 2025-11-03T16:32:55.4790051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4790415Z outputs = self.electra( 2025-11-03T16:32:55.4790771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4791140Z hidden_states = self.encoder( 2025-11-03T16:32:55.4791513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4791889Z layer_outputs = layer_module( 2025-11-03T16:32:55.4792214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4792561Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4792922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4793296Z return func(*args, **kwargs) 2025-11-03T16:32:55.4793657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4794146Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4794542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4794932Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4795348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4795835Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4796248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4796652Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4797012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4797332Z return self.act(input) 2025-11-03T16:32:55.4797437Z 2025-11-03T16:32:55.4797532Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4797869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4798170Z res = mod(**inputs) 2025-11-03T16:32:55.4798522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4798894Z outputs = self.electra( 2025-11-03T16:32:55.4799260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4799625Z hidden_states = self.encoder( 2025-11-03T16:32:55.4799986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4800350Z layer_outputs = layer_module( 2025-11-03T16:32:55.4800663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4801003Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4801388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4801737Z return func(*args, **kwargs) 2025-11-03T16:32:55.4802092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4802462Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4802833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4803198Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4803591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4804035Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4804455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4804832Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4804969Z 2025-11-03T16:32:55.4805067Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4805401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4805691Z res = mod(**inputs) 2025-11-03T16:32:55.4806037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4806431Z outputs = self.electra( 2025-11-03T16:32:55.4806783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4807147Z hidden_states = self.encoder( 2025-11-03T16:32:55.4807498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4807865Z layer_outputs = layer_module( 2025-11-03T16:32:55.4808190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4808529Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4808893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4809240Z return func(*args, **kwargs) 2025-11-03T16:32:55.4809598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4809977Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4810333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4810672Z return func(*args, **kwargs) 2025-11-03T16:32:55.4811027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4811394Z self_outputs = self.self( 2025-11-03T16:32:55.4811735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4812078Z return func(*args, **kwargs) 2025-11-03T16:32:55.4812424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4812798Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4812934Z 2025-11-03T16:32:55.4813030Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4813476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4813780Z res = mod(**inputs) 2025-11-03T16:32:55.4814130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4814496Z outputs = self.electra( 2025-11-03T16:32:55.4814914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4815286Z hidden_states = self.encoder( 2025-11-03T16:32:55.4815637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4816003Z layer_outputs = layer_module( 2025-11-03T16:32:55.4816332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4816674Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4817034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4817382Z return func(*args, **kwargs) 2025-11-03T16:32:55.4817743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4818124Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4818487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4818832Z return func(*args, **kwargs) 2025-11-03T16:32:55.4819191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4819584Z self_outputs = self.self( 2025-11-03T16:32:55.4819923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4820270Z return func(*args, **kwargs) 2025-11-03T16:32:55.4820618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4820990Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4821124Z 2025-11-03T16:32:55.4821222Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4821558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4821880Z res = mod(**inputs) 2025-11-03T16:32:55.4822219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4822587Z outputs = self.electra( 2025-11-03T16:32:55.4822939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4823299Z hidden_states = self.encoder( 2025-11-03T16:32:55.4823647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4824007Z layer_outputs = layer_module( 2025-11-03T16:32:55.4824334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4824672Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4825027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4825367Z return func(*args, **kwargs) 2025-11-03T16:32:55.4825720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4826095Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4826449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4826787Z return func(*args, **kwargs) 2025-11-03T16:32:55.4827137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4827503Z self_outputs = self.self( 2025-11-03T16:32:55.4827899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4828245Z return func(*args, **kwargs) 2025-11-03T16:32:55.4828599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4828978Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4829112Z 2025-11-03T16:32:55.4829192Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4829396Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4829622Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4829961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4830271Z res = mod(**inputs) 2025-11-03T16:32:55.4830631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4831010Z outputs = self.electra( 2025-11-03T16:32:55.4831361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4831731Z hidden_states = self.encoder( 2025-11-03T16:32:55.4832103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4832496Z layer_outputs = layer_module( 2025-11-03T16:32:55.4832828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4833172Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4833535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4833891Z return func(*args, **kwargs) 2025-11-03T16:32:55.4834339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4834748Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4835141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4835541Z return func(*args, **kwargs) 2025-11-03T16:32:55.4835967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4836409Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4836834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4837224Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4837366Z 2025-11-03T16:32:55.4837467Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4837821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4838132Z res = mod(**inputs) 2025-11-03T16:32:55.4838482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4838865Z outputs = self.electra( 2025-11-03T16:32:55.4839224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4839608Z hidden_states = self.encoder( 2025-11-03T16:32:55.4839971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4840344Z layer_outputs = layer_module( 2025-11-03T16:32:55.4840674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4841024Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4841422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4841772Z return func(*args, **kwargs) 2025-11-03T16:32:55.4842140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4842527Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4842911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4843286Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4843684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4844135Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4844558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4844945Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4845075Z 2025-11-03T16:32:55.4845182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4845523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4845827Z res = mod(**inputs) 2025-11-03T16:32:55.4846183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4846583Z outputs = self.electra( 2025-11-03T16:32:55.4846940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4847314Z hidden_states = self.encoder( 2025-11-03T16:32:55.4847681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4848056Z layer_outputs = layer_module( 2025-11-03T16:32:55.4848382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4848729Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4849084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4849430Z return func(*args, **kwargs) 2025-11-03T16:32:55.4849788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4850166Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4850532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4850892Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4851292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4851730Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4852136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4852540Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4852900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4853222Z return self.act(input) 2025-11-03T16:32:55.4853328Z 2025-11-03T16:32:55.4853432Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4853758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4854057Z res = mod(**inputs) 2025-11-03T16:32:55.4854436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4854812Z outputs = self.electra( 2025-11-03T16:32:55.4855185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4855548Z hidden_states = self.encoder( 2025-11-03T16:32:55.4855913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4856284Z layer_outputs = layer_module( 2025-11-03T16:32:55.4856614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4856952Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4857311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4857664Z return func(*args, **kwargs) 2025-11-03T16:32:55.4858030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4858426Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4858799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4859168Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4859582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4860039Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4860466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4860838Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4860974Z 2025-11-03T16:32:55.4861074Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4861452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4861785Z res = mod(**inputs) 2025-11-03T16:32:55.4862131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4862491Z outputs = self.electra( 2025-11-03T16:32:55.4862842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4863205Z hidden_states = self.encoder( 2025-11-03T16:32:55.4863562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4863921Z layer_outputs = layer_module( 2025-11-03T16:32:55.4864245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4864589Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4864950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4865301Z return func(*args, **kwargs) 2025-11-03T16:32:55.4865647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4866089Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4866457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4866815Z return func(*args, **kwargs) 2025-11-03T16:32:55.4867176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4867553Z self_outputs = self.self( 2025-11-03T16:32:55.4867938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4868291Z return func(*args, **kwargs) 2025-11-03T16:32:55.4868656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4869034Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4869174Z 2025-11-03T16:32:55.4869274Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4869617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4869931Z res = mod(**inputs) 2025-11-03T16:32:55.4870287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4870656Z outputs = self.electra( 2025-11-03T16:32:55.4871017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4871420Z hidden_states = self.encoder( 2025-11-03T16:32:55.4871786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4872156Z layer_outputs = layer_module( 2025-11-03T16:32:55.4872488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4872853Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4890186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4890607Z return func(*args, **kwargs) 2025-11-03T16:32:55.4891020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4891442Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4891853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4892211Z return func(*args, **kwargs) 2025-11-03T16:32:55.4892679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4893053Z self_outputs = self.self( 2025-11-03T16:32:55.4893392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4893747Z return func(*args, **kwargs) 2025-11-03T16:32:55.4894112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4894488Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4894621Z 2025-11-03T16:32:55.4894733Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4895082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4895393Z res = mod(**inputs) 2025-11-03T16:32:55.4895751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4896127Z outputs = self.electra( 2025-11-03T16:32:55.4896485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4896850Z hidden_states = self.encoder( 2025-11-03T16:32:55.4897213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4897575Z layer_outputs = layer_module( 2025-11-03T16:32:55.4897902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4898234Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4898670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4899023Z return func(*args, **kwargs) 2025-11-03T16:32:55.4899386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4899774Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4900138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4900489Z return func(*args, **kwargs) 2025-11-03T16:32:55.4900852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4901224Z self_outputs = self.self( 2025-11-03T16:32:55.4901559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4901914Z return func(*args, **kwargs) 2025-11-03T16:32:55.4902275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4902661Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4902793Z 2025-11-03T16:32:55.4902880Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4903078Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4903335Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4903679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4903987Z res = mod(**inputs) 2025-11-03T16:32:55.4904338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4904713Z outputs = self.electra( 2025-11-03T16:32:55.4904969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4905039Z hidden_states = self.encoder( 2025-11-03T16:32:55.4905287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4905373Z layer_outputs = layer_module( 2025-11-03T16:32:55.4905594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4905681Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4905905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4905980Z return func(*args, **kwargs) 2025-11-03T16:32:55.4906223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4906300Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4906547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4906613Z return func(*args, **kwargs) 2025-11-03T16:32:55.4906867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4906987Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4907227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4907314Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4907318Z 2025-11-03T16:32:55.4907419Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4907612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4907672Z res = mod(**inputs) 2025-11-03T16:32:55.4907971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4908038Z outputs = self.electra( 2025-11-03T16:32:55.4908288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4908367Z hidden_states = self.encoder( 2025-11-03T16:32:55.4908616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4908696Z layer_outputs = layer_module( 2025-11-03T16:32:55.4908911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4908988Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4909231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4909301Z return func(*args, **kwargs) 2025-11-03T16:32:55.4909558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4909644Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4909904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4909998Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4910279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4910405Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4910655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4910744Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4910751Z 2025-11-03T16:32:55.4910851Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4911043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4911131Z res = mod(**inputs) 2025-11-03T16:32:55.4911385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4911461Z outputs = self.electra( 2025-11-03T16:32:55.4911713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4911788Z hidden_states = self.encoder( 2025-11-03T16:32:55.4912037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4912104Z layer_outputs = layer_module( 2025-11-03T16:32:55.4912329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4912404Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4912645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4912711Z return func(*args, **kwargs) 2025-11-03T16:32:55.4912964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4913053Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4913457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4913545Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4913826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4914087Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4914358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4914469Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4914704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4914779Z return self.act(input) 2025-11-03T16:32:55.4914783Z 2025-11-03T16:32:55.4914897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4915099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4915164Z res = mod(**inputs) 2025-11-03T16:32:55.4915438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4915506Z outputs = self.electra( 2025-11-03T16:32:55.4915762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4915831Z hidden_states = self.encoder( 2025-11-03T16:32:55.4916077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4916153Z layer_outputs = layer_module( 2025-11-03T16:32:55.4916390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4916473Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4916706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4916779Z return func(*args, **kwargs) 2025-11-03T16:32:55.4917028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4917110Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4917361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4917459Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4917744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4917877Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4918123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4918209Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4918212Z 2025-11-03T16:32:55.4918313Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4918516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4918578Z res = mod(**inputs) 2025-11-03T16:32:55.4918836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4918905Z outputs = self.electra( 2025-11-03T16:32:55.4919154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4919233Z hidden_states = self.encoder( 2025-11-03T16:32:55.4919482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4919555Z layer_outputs = layer_module( 2025-11-03T16:32:55.4919767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4919839Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4920109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4920177Z return func(*args, **kwargs) 2025-11-03T16:32:55.4920432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4920513Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4920745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4920822Z return func(*args, **kwargs) 2025-11-03T16:32:55.4921069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4921147Z self_outputs = self.self( 2025-11-03T16:32:55.4921378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4921454Z return func(*args, **kwargs) 2025-11-03T16:32:55.4921703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4921785Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4921789Z 2025-11-03T16:32:55.4921895Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4922082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4922170Z res = mod(**inputs) 2025-11-03T16:32:55.4922422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4922488Z outputs = self.electra( 2025-11-03T16:32:55.4922747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4922816Z hidden_states = self.encoder( 2025-11-03T16:32:55.4923077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4923144Z layer_outputs = layer_module( 2025-11-03T16:32:55.4923368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4923450Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4923680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4923756Z return func(*args, **kwargs) 2025-11-03T16:32:55.4924002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4924086Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4924316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4924384Z return func(*args, **kwargs) 2025-11-03T16:32:55.4924637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4924706Z self_outputs = self.self( 2025-11-03T16:32:55.4924945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4925011Z return func(*args, **kwargs) 2025-11-03T16:32:55.4925258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4925342Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4925346Z 2025-11-03T16:32:55.4925444Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4925640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4925703Z res = mod(**inputs) 2025-11-03T16:32:55.4925988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4926057Z outputs = self.electra( 2025-11-03T16:32:55.4926309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4926384Z hidden_states = self.encoder( 2025-11-03T16:32:55.4926637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4926709Z layer_outputs = layer_module( 2025-11-03T16:32:55.4926915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4926985Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4927216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4927282Z return func(*args, **kwargs) 2025-11-03T16:32:55.4927525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4927603Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4927822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4927909Z return func(*args, **kwargs) 2025-11-03T16:32:55.4928154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4928225Z self_outputs = self.self( 2025-11-03T16:32:55.4928451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4928523Z return func(*args, **kwargs) 2025-11-03T16:32:55.4928768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4928842Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4928879Z 2025-11-03T16:32:55.4928965Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4929040Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4929143Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4929325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4929386Z res = mod(**inputs) 2025-11-03T16:32:55.4929639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4929703Z outputs = self.electra( 2025-11-03T16:32:55.4929948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4930013Z hidden_states = self.encoder( 2025-11-03T16:32:55.4930258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4930333Z layer_outputs = layer_module( 2025-11-03T16:32:55.4930538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4930617Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4930843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4930913Z return func(*args, **kwargs) 2025-11-03T16:32:55.4931156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4931233Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4931496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4931562Z return func(*args, **kwargs) 2025-11-03T16:32:55.4931810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4931934Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4932176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4932266Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4932269Z 2025-11-03T16:32:55.4932365Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4932556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4932617Z res = mod(**inputs) 2025-11-03T16:32:55.4932861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4932933Z outputs = self.electra( 2025-11-03T16:32:55.4933175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4933251Z hidden_states = self.encoder( 2025-11-03T16:32:55.4933492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4933580Z layer_outputs = layer_module( 2025-11-03T16:32:55.4933788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4933861Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4934092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4934157Z return func(*args, **kwargs) 2025-11-03T16:32:55.4934404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4934483Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4934750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4934829Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4935099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4935221Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4935462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4935545Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4935548Z 2025-11-03T16:32:55.4935643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4935826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4935894Z res = mod(**inputs) 2025-11-03T16:32:55.4936137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4936208Z outputs = self.electra( 2025-11-03T16:32:55.4936449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4936515Z hidden_states = self.encoder( 2025-11-03T16:32:55.4936764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4936829Z layer_outputs = layer_module( 2025-11-03T16:32:55.4937044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4937144Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4937376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4937444Z return func(*args, **kwargs) 2025-11-03T16:32:55.4937687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4937773Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4938015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4938093Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4938367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4938479Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4938731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4938840Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4939048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4939115Z return self.act(input) 2025-11-03T16:32:55.4939119Z 2025-11-03T16:32:55.4939222Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4939425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4939486Z res = mod(**inputs) 2025-11-03T16:32:55.4939740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4939804Z outputs = self.electra( 2025-11-03T16:32:55.4940057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4940122Z hidden_states = self.encoder( 2025-11-03T16:32:55.4940365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4940456Z layer_outputs = layer_module( 2025-11-03T16:32:55.4940660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4940741Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4940963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4941031Z return func(*args, **kwargs) 2025-11-03T16:32:55.4941289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4941368Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4941608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4941687Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4941959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4942092Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4942334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4942418Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4942421Z 2025-11-03T16:32:55.4942517Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4942704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4942771Z res = mod(**inputs) 2025-11-03T16:32:55.4943042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4943116Z outputs = self.electra( 2025-11-03T16:32:55.4943367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4943433Z hidden_states = self.encoder( 2025-11-03T16:32:55.4943682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4943749Z layer_outputs = layer_module( 2025-11-03T16:32:55.4943963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4944035Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4944266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4944335Z return func(*args, **kwargs) 2025-11-03T16:32:55.4944572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4944660Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4944883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4944970Z return func(*args, **kwargs) 2025-11-03T16:32:55.4945214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4945280Z self_outputs = self.self( 2025-11-03T16:32:55.4945520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4945585Z return func(*args, **kwargs) 2025-11-03T16:32:55.4945839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4945915Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4945919Z 2025-11-03T16:32:55.4946037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4946223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4946281Z res = mod(**inputs) 2025-11-03T16:32:55.4946533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4946598Z outputs = self.electra( 2025-11-03T16:32:55.4946843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4946908Z hidden_states = self.encoder( 2025-11-03T16:32:55.4947146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4947220Z layer_outputs = layer_module( 2025-11-03T16:32:55.4947426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4947508Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4947732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4947798Z return func(*args, **kwargs) 2025-11-03T16:32:55.4948049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4948127Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4948361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4948426Z return func(*args, **kwargs) 2025-11-03T16:32:55.4948702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4948769Z self_outputs = self.self( 2025-11-03T16:32:55.4948993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4949065Z return func(*args, **kwargs) 2025-11-03T16:32:55.4949306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4949390Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4949393Z 2025-11-03T16:32:55.4949489Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4949671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4949738Z res = mod(**inputs) 2025-11-03T16:32:55.4949990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4950063Z outputs = self.electra( 2025-11-03T16:32:55.4950314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4950383Z hidden_states = self.encoder( 2025-11-03T16:32:55.4950636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4950723Z layer_outputs = layer_module( 2025-11-03T16:32:55.4950944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4951018Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4951252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4951316Z return func(*args, **kwargs) 2025-11-03T16:32:55.4951562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4951648Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4951895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4951969Z return func(*args, **kwargs) 2025-11-03T16:32:55.4952218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4952284Z self_outputs = self.self( 2025-11-03T16:32:55.4952524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4952587Z return func(*args, **kwargs) 2025-11-03T16:32:55.4952843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4952920Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4952924Z 2025-11-03T16:32:55.4953008Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4953084Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4953182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4953380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4953442Z res = mod(**inputs) 2025-11-03T16:32:55.4953706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4953771Z outputs = self.electra( 2025-11-03T16:32:55.4954119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4954204Z hidden_states = self.encoder( 2025-11-03T16:32:55.4954508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4954588Z layer_outputs = layer_module( 2025-11-03T16:32:55.4954812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4954894Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4955173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4955242Z return func(*args, **kwargs) 2025-11-03T16:32:55.4955497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4955573Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4955805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4955879Z return func(*args, **kwargs) 2025-11-03T16:32:55.4956128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4956261Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4956518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4956605Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4956630Z 2025-11-03T16:32:55.4956730Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4956917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4956989Z res = mod(**inputs) 2025-11-03T16:32:55.4957242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4957314Z outputs = self.electra( 2025-11-03T16:32:55.4957565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4957634Z hidden_states = self.encoder( 2025-11-03T16:32:55.4957905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4957972Z layer_outputs = layer_module( 2025-11-03T16:32:55.4958195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4958271Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4958513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4958579Z return func(*args, **kwargs) 2025-11-03T16:32:55.4958831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4958921Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4959170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4959254Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4959542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4959660Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4959926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4960002Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4960005Z 2025-11-03T16:32:55.4960111Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4960303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4960398Z res = mod(**inputs) 2025-11-03T16:32:55.4960654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4960721Z outputs = self.electra( 2025-11-03T16:32:55.4960977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4961043Z hidden_states = self.encoder( 2025-11-03T16:32:55.4961298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4961365Z layer_outputs = layer_module( 2025-11-03T16:32:55.4961574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4961655Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4961889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4961962Z return func(*args, **kwargs) 2025-11-03T16:32:55.4962211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4962293Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4962546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4962657Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4963058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4963174Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4963433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4963545Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4963755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4963849Z return self.act(input) 2025-11-03T16:32:55.4963853Z 2025-11-03T16:32:55.4963951Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4964145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4964208Z res = mod(**inputs) 2025-11-03T16:32:55.4964467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4964535Z outputs = self.electra( 2025-11-03T16:32:55.4964785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4964862Z hidden_states = self.encoder( 2025-11-03T16:32:55.4965112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4965186Z layer_outputs = layer_module( 2025-11-03T16:32:55.4965396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4965470Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4965705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4965774Z return func(*args, **kwargs) 2025-11-03T16:32:55.4966028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4966106Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4966348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4967182Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4967468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4967605Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4967853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4967940Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4967943Z 2025-11-03T16:32:55.4968043Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4968228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4968298Z res = mod(**inputs) 2025-11-03T16:32:55.4968551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4968626Z outputs = self.electra( 2025-11-03T16:32:55.4968881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4968954Z hidden_states = self.encoder( 2025-11-03T16:32:55.4969225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4969310Z layer_outputs = layer_module( 2025-11-03T16:32:55.4969527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4969601Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4969842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4969911Z return func(*args, **kwargs) 2025-11-03T16:32:55.4970168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4970258Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4970492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4970585Z return func(*args, **kwargs) 2025-11-03T16:32:55.4970837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4970907Z self_outputs = self.self( 2025-11-03T16:32:55.4971150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4971215Z return func(*args, **kwargs) 2025-11-03T16:32:55.4971476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4971556Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4971563Z 2025-11-03T16:32:55.4971662Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4971862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4971927Z res = mod(**inputs) 2025-11-03T16:32:55.4972193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4972260Z outputs = self.electra( 2025-11-03T16:32:55.4972520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4972589Z hidden_states = self.encoder( 2025-11-03T16:32:55.4972842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4972919Z layer_outputs = layer_module( 2025-11-03T16:32:55.4973178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4973262Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4973502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4973568Z return func(*args, **kwargs) 2025-11-03T16:32:55.4973834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4973913Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4974155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4974222Z return func(*args, **kwargs) 2025-11-03T16:32:55.4974477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4974552Z self_outputs = self.self( 2025-11-03T16:32:55.4974789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4974865Z return func(*args, **kwargs) 2025-11-03T16:32:55.4975119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.4975201Z key_layer = self.key(current_states) 2025-11-03T16:32:55.4975220Z 2025-11-03T16:32:55.4975321Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4975512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4975582Z res = mod(**inputs) 2025-11-03T16:32:55.4975842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4975915Z outputs = self.electra( 2025-11-03T16:32:55.4976172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4976240Z hidden_states = self.encoder( 2025-11-03T16:32:55.4976515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4976585Z layer_outputs = layer_module( 2025-11-03T16:32:55.4976807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4976886Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4977127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4977194Z return func(*args, **kwargs) 2025-11-03T16:32:55.4977447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4977537Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4977770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4977845Z return func(*args, **kwargs) 2025-11-03T16:32:55.4978100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4978167Z self_outputs = self.self( 2025-11-03T16:32:55.4978411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4978479Z return func(*args, **kwargs) 2025-11-03T16:32:55.4978740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.4978817Z value_layer = self.value(current_states) 2025-11-03T16:32:55.4978820Z 2025-11-03T16:32:55.4978897Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4979014Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.4979115Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4979316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4979380Z res = mod(**inputs) 2025-11-03T16:32:55.4979639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4979712Z outputs = self.electra( 2025-11-03T16:32:55.4979965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4980040Z hidden_states = self.encoder( 2025-11-03T16:32:55.4980293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4980366Z layer_outputs = layer_module( 2025-11-03T16:32:55.4980584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4980662Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4980908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4980976Z return func(*args, **kwargs) 2025-11-03T16:32:55.4981234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4981326Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4981606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4981679Z return func(*args, **kwargs) 2025-11-03T16:32:55.4981924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.4982053Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.4982298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.4982397Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4982400Z 2025-11-03T16:32:55.4982498Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4982688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4982757Z res = mod(**inputs) 2025-11-03T16:32:55.4983010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4983083Z outputs = self.electra( 2025-11-03T16:32:55.4983395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4983466Z hidden_states = self.encoder( 2025-11-03T16:32:55.4983735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4983805Z layer_outputs = layer_module( 2025-11-03T16:32:55.4984039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4984110Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4984345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4984419Z return func(*args, **kwargs) 2025-11-03T16:32:55.4984670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4984756Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4985040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4985114Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4985393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4985518Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4985765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.4985850Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4985854Z 2025-11-03T16:32:55.4985950Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4986141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4986209Z res = mod(**inputs) 2025-11-03T16:32:55.4986462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4986534Z outputs = self.electra( 2025-11-03T16:32:55.4986779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4986852Z hidden_states = self.encoder( 2025-11-03T16:32:55.4987097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4987179Z layer_outputs = layer_module( 2025-11-03T16:32:55.4987398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4987470Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4987705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4987771Z return func(*args, **kwargs) 2025-11-03T16:32:55.4988018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4988106Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4988363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4988444Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4988723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.4988844Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.4989091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.4989200Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.4989415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.4989484Z return self.act(input) 2025-11-03T16:32:55.4989487Z 2025-11-03T16:32:55.4989594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4989784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4989844Z res = mod(**inputs) 2025-11-03T16:32:55.4990103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4990170Z outputs = self.electra( 2025-11-03T16:32:55.4990425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4990491Z hidden_states = self.encoder( 2025-11-03T16:32:55.4990736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4990840Z layer_outputs = layer_module( 2025-11-03T16:32:55.4991050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4991133Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4991362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4991436Z return func(*args, **kwargs) 2025-11-03T16:32:55.4991683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.4991759Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.4992008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.4992080Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.4992366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.4992498Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.4992755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.4992832Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.4992850Z 2025-11-03T16:32:55.4992956Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4993141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4993202Z res = mod(**inputs) 2025-11-03T16:32:55.4993456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4993520Z outputs = self.electra( 2025-11-03T16:32:55.4993774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4993840Z hidden_states = self.encoder( 2025-11-03T16:32:55.4994231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4994316Z layer_outputs = layer_module( 2025-11-03T16:32:55.4994539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4994627Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4994874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4994943Z return func(*args, **kwargs) 2025-11-03T16:32:55.4995218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4995298Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4995546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4995616Z return func(*args, **kwargs) 2025-11-03T16:32:55.4995882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4995947Z self_outputs = self.self( 2025-11-03T16:32:55.4996172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4996246Z return func(*args, **kwargs) 2025-11-03T16:32:55.4996488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.4996570Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.4996574Z 2025-11-03T16:32:55.4996670Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.4996896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.4996966Z res = mod(**inputs) 2025-11-03T16:32:55.4997209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.4997278Z outputs = self.electra( 2025-11-03T16:32:55.4997518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.4997590Z hidden_states = self.encoder( 2025-11-03T16:32:55.4997828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.4997893Z layer_outputs = layer_module( 2025-11-03T16:32:55.4998102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.4998177Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.4998403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4998468Z return func(*args, **kwargs) 2025-11-03T16:32:55.4998706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.4998787Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.4999028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4999096Z return func(*args, **kwargs) 2025-11-03T16:32:55.4999335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.4999400Z self_outputs = self.self( 2025-11-03T16:32:55.4999631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.4999694Z return func(*args, **kwargs) 2025-11-03T16:32:55.4999938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.5000024Z key_layer = self.key(current_states) 2025-11-03T16:32:55.5000028Z 2025-11-03T16:32:55.5000129Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5000311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5000370Z res = mod(**inputs) 2025-11-03T16:32:55.5000622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5000685Z outputs = self.electra( 2025-11-03T16:32:55.5000929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5000997Z hidden_states = self.encoder( 2025-11-03T16:32:55.5001238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5001311Z layer_outputs = layer_module( 2025-11-03T16:32:55.5001516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5001592Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5001816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5001880Z return func(*args, **kwargs) 2025-11-03T16:32:55.5002129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.5002203Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.5002464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5002527Z return func(*args, **kwargs) 2025-11-03T16:32:55.5002773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.5002839Z self_outputs = self.self( 2025-11-03T16:32:55.5003062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5003135Z return func(*args, **kwargs) 2025-11-03T16:32:55.5003379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.5003458Z value_layer = self.value(current_states) 2025-11-03T16:32:55.5003461Z 2025-11-03T16:32:55.5003534Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.5003606Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.5003709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5003889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5003956Z res = mod(**inputs) 2025-11-03T16:32:55.5004199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5004263Z outputs = self.electra( 2025-11-03T16:32:55.5004504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5004584Z hidden_states = self.encoder( 2025-11-03T16:32:55.5004837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5004902Z layer_outputs = layer_module( 2025-11-03T16:32:55.5005118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5005193Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5005423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5005508Z return func(*args, **kwargs) 2025-11-03T16:32:55.5005748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.5005830Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.5006057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5006121Z return func(*args, **kwargs) 2025-11-03T16:32:55.5006369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.5006487Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.5006737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.5006813Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.5006817Z 2025-11-03T16:32:55.5006918Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5007100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5007159Z res = mod(**inputs) 2025-11-03T16:32:55.5007413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5007475Z outputs = self.electra( 2025-11-03T16:32:55.5007726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5007791Z hidden_states = self.encoder( 2025-11-03T16:32:55.5008072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5008147Z layer_outputs = layer_module( 2025-11-03T16:32:55.5008350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5008431Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5008656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5008722Z return func(*args, **kwargs) 2025-11-03T16:32:55.5008968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.5009044Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.5009292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.5009364Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.5009644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.5009758Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.5009999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.5010083Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.5010101Z 2025-11-03T16:32:55.5010195Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5010381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5010440Z res = mod(**inputs) 2025-11-03T16:32:55.5010684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5010755Z outputs = self.electra( 2025-11-03T16:32:55.5010996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5011082Z hidden_states = self.encoder( 2025-11-03T16:32:55.5011322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5011393Z layer_outputs = layer_module( 2025-11-03T16:32:55.5011599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5011670Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5011899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5011964Z return func(*args, **kwargs) 2025-11-03T16:32:55.5012211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.5012289Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.5012527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.5012606Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.5012877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.5012994Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.5013376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.5013496Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.5013700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.5013766Z return self.act(input) 2025-11-03T16:32:55.5013835Z 2025-11-03T16:32:55.5013938Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5014121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5014191Z res = mod(**inputs) 2025-11-03T16:32:55.5014435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5014500Z outputs = self.electra( 2025-11-03T16:32:55.5014749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5014813Z hidden_states = self.encoder( 2025-11-03T16:32:55.5015060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5015125Z layer_outputs = layer_module( 2025-11-03T16:32:55.5015332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5015413Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5015637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5015711Z return func(*args, **kwargs) 2025-11-03T16:32:55.5015952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.5016062Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.5016302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.5016376Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.5016657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.5016787Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.5017038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.5017143Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.5017147Z 2025-11-03T16:32:55.5017244Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5017435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5017496Z res = mod(**inputs) 2025-11-03T16:32:55.5017746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5017808Z outputs = self.electra( 2025-11-03T16:32:55.5018054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5018120Z hidden_states = self.encoder( 2025-11-03T16:32:55.5018363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5018444Z layer_outputs = layer_module( 2025-11-03T16:32:55.5018647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5018727Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5018950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5019014Z return func(*args, **kwargs) 2025-11-03T16:32:55.5019261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.5019336Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.5019594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5019659Z return func(*args, **kwargs) 2025-11-03T16:32:55.5019899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.5019971Z self_outputs = self.self( 2025-11-03T16:32:55.5020192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5020266Z return func(*args, **kwargs) 2025-11-03T16:32:55.5020504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 240, in forward 2025-11-03T16:32:55.5020586Z query_layer = self.query(hidden_states) 2025-11-03T16:32:55.5020589Z 2025-11-03T16:32:55.5020683Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5020865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5020934Z res = mod(**inputs) 2025-11-03T16:32:55.5021178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5021251Z outputs = self.electra( 2025-11-03T16:32:55.5021488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5021552Z hidden_states = self.encoder( 2025-11-03T16:32:55.5021814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5021880Z layer_outputs = layer_module( 2025-11-03T16:32:55.5022089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5022159Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5022392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5022455Z return func(*args, **kwargs) 2025-11-03T16:32:55.5022694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.5022795Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.5023018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5023090Z return func(*args, **kwargs) 2025-11-03T16:32:55.5023331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.5023395Z self_outputs = self.self( 2025-11-03T16:32:55.5023624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5023688Z return func(*args, **kwargs) 2025-11-03T16:32:55.5023934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-11-03T16:32:55.5024006Z key_layer = self.key(current_states) 2025-11-03T16:32:55.5024011Z 2025-11-03T16:32:55.5024106Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5024295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5024355Z res = mod(**inputs) 2025-11-03T16:32:55.5024612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5024674Z outputs = self.electra( 2025-11-03T16:32:55.5024919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5024984Z hidden_states = self.encoder( 2025-11-03T16:32:55.5025252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5025326Z layer_outputs = layer_module( 2025-11-03T16:32:55.5025531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5025609Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5025835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5025900Z return func(*args, **kwargs) 2025-11-03T16:32:55.5026148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.5026222Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.5026455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5026519Z return func(*args, **kwargs) 2025-11-03T16:32:55.5026762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-11-03T16:32:55.5026833Z self_outputs = self.self( 2025-11-03T16:32:55.5027057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5027124Z return func(*args, **kwargs) 2025-11-03T16:32:55.5027381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-11-03T16:32:55.5027460Z value_layer = self.value(current_states) 2025-11-03T16:32:55.5027463Z 2025-11-03T16:32:55.5027538Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.5027611Z cudagraph partition due to non gpu ops 2025-11-03T16:32:55.5027714Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5027898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5027965Z res = mod(**inputs) 2025-11-03T16:32:55.5028212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5028291Z outputs = self.electra( 2025-11-03T16:32:55.5028539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5028605Z hidden_states = self.encoder( 2025-11-03T16:32:55.5028851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5028916Z layer_outputs = layer_module( 2025-11-03T16:32:55.5029118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5029196Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5029421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5029490Z return func(*args, **kwargs) 2025-11-03T16:32:55.5029729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-11-03T16:32:55.5029813Z self_attention_outputs = self.attention( 2025-11-03T16:32:55.5030035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5030101Z return func(*args, **kwargs) 2025-11-03T16:32:55.5030348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-11-03T16:32:55.5030468Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:32:55.5030714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-11-03T16:32:55.5030817Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.5030821Z 2025-11-03T16:32:55.5030917Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5031105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5031166Z res = mod(**inputs) 2025-11-03T16:32:55.5031417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5031483Z outputs = self.electra( 2025-11-03T16:32:55.5031730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5031794Z hidden_states = self.encoder( 2025-11-03T16:32:55.5032034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5032106Z layer_outputs = layer_module( 2025-11-03T16:32:55.5032312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5032391Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5032616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5032679Z return func(*args, **kwargs) 2025-11-03T16:32:55.5032942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.5033019Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.5033263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.5033334Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.5033606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.5033729Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.5034047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-11-03T16:32:55.5034158Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.5034162Z 2025-11-03T16:32:55.5034263Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5034469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5034532Z res = mod(**inputs) 2025-11-03T16:32:55.5034799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5034876Z outputs = self.electra( 2025-11-03T16:32:55.5035150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5035226Z hidden_states = self.encoder( 2025-11-03T16:32:55.5035469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5035535Z layer_outputs = layer_module( 2025-11-03T16:32:55.5035763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5035843Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5036092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5036161Z return func(*args, **kwargs) 2025-11-03T16:32:55.5036427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.5036510Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.5036797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.5036881Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.5037172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-11-03T16:32:55.5037295Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:32:55.5037554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-11-03T16:32:55.5037664Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:32:55.5037884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:32:55.5037955Z return self.act(input) 2025-11-03T16:32:55.5037958Z 2025-11-03T16:32:55.5038068Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5038265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5038335Z res = mod(**inputs) 2025-11-03T16:32:55.5038593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-11-03T16:32:55.5038660Z outputs = self.electra( 2025-11-03T16:32:55.5038922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-11-03T16:32:55.5039015Z hidden_states = self.encoder( 2025-11-03T16:32:55.5039284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-11-03T16:32:55.5039350Z layer_outputs = layer_module( 2025-11-03T16:32:55.5039576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:32:55.5039659Z return super().__call__(*args, **kwargs) 2025-11-03T16:32:55.5039908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:32:55.5039998Z return func(*args, **kwargs) 2025-11-03T16:32:55.5040255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-11-03T16:32:55.5040335Z layer_output = apply_chunking_to_forward( 2025-11-03T16:32:55.5040594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:32:55.5040667Z return forward_fn(*input_tensors) 2025-11-03T16:32:55.5040960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-11-03T16:32:55.5041089Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:32:55.5041358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-11-03T16:32:55.5041438Z hidden_states = self.dense(hidden_states) 2025-11-03T16:32:55.5041443Z 2025-11-03T16:32:55.5041542Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5041740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5041801Z res = mod(**inputs) 2025-11-03T16:32:55.5042066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-11-03T16:32:55.5042242Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-11-03T16:32:55.5042506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 630, in forward 2025-11-03T16:32:55.5042607Z hidden_states = self.dense(generator_hidden_states) 2025-11-03T16:32:55.5042611Z 2025-11-03T16:32:55.5042738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5042937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5043002Z res = mod(**inputs) 2025-11-03T16:32:55.5043270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-11-03T16:32:55.5043443Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-11-03T16:32:55.5043447Z 2025-11-03T16:32:55.5043545Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:32:55.5043742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:32:55.5043802Z res = mod(**inputs) 2025-11-03T16:32:55.5044066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1554, in forward 2025-11-03T16:32:55.5044134Z lm_loss = self.loss_function( 2025-11-03T16:32:55.5044366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-11-03T16:32:55.5044530Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-11-03T16:32:55.5044763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-11-03T16:32:55.5044966Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-11-03T16:32:55.5044970Z 2025-11-03T16:33:05.1612411Z Compilation time (from dynamo_timed): 16.72164073 2025-11-03T16:33:05.1723324Z pass 2025-11-03T16:33:05.1727438Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:05.1731636Z TIMING: _recursive_pre_grad_passes:0.00656 _recursive_joint_graph_passes:0.70699 _recursive_post_grad_passes:0.06111 async_compile.wait:0.7396 code_gen:9.11628 inductor_compile:10.36573 backend_compile:13.69862 gc:0.00131 entire_frame_compile:16.72164 total_wall_time:16.72164 2025-11-03T16:33:05.1733085Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:8565 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:2529 2025-11-03T16:33:05.1733567Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-11-03T16:33:07.3107262Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:33:07.3108291Z import pynvml # type: ignore[import] 2025-11-03T16:33:10.8020964Z 2025-11-03T16:33:11.9350868Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:33:11.9351617Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:33:11.9352025Z cpu eval GPT2ForSequenceClassification 2025-11-03T16:33:12.7154589Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:13.0687882Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:13.4114472Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:19.7557044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7558827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7564296Z res = mod(**inputs) 2025-11-03T16:33:19.7566675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7567243Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7572856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:33:19.7573950Z causal_mask = create_causal_mask( 2025-11-03T16:33:19.7574362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:33:19.7574892Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:33:19.7575438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:33:19.7575904Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:33:19.7576373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 654, in find_packed_sequence_indices 2025-11-03T16:33:19.7576888Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-11-03T16:33:19.7577123Z 2025-11-03T16:33:19.7577219Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7577476Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7577856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7578199Z res = mod(**inputs) 2025-11-03T16:33:19.7578578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1396, in forward 2025-11-03T16:33:19.7579105Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-11-03T16:33:19.7579296Z 2025-11-03T16:33:19.7579409Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7579790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7580131Z res = mod(**inputs) 2025-11-03T16:33:19.7580496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7580913Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7581318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7581763Z outputs = block( 2025-11-03T16:33:19.7582188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7582555Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7582950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7583336Z return func(*args, **kwargs) 2025-11-03T16:33:19.7583720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7584130Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7584525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7584907Z return func(*args, **kwargs) 2025-11-03T16:33:19.7585286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7585803Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7586274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7586683Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7586868Z 2025-11-03T16:33:19.7586975Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7587340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7587664Z res = mod(**inputs) 2025-11-03T16:33:19.7588107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7588521Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7588921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:33:19.7589326Z causal_mask = create_causal_mask( 2025-11-03T16:33:19.7589716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:33:19.7590222Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:33:19.7590753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:33:19.7591220Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:33:19.7591671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-11-03T16:33:19.7592140Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-11-03T16:33:19.7592349Z 2025-11-03T16:33:19.7592430Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7592664Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7593021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7593392Z res = mod(**inputs) 2025-11-03T16:33:19.7593762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7594295Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7594702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:33:19.7595095Z causal_mask = create_causal_mask( 2025-11-03T16:33:19.7595492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:33:19.7595985Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:33:19.7596521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:33:19.7596967Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:33:19.7597418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-11-03T16:33:19.7597853Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-11-03T16:33:19.7598018Z 2025-11-03T16:33:19.7598128Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7598476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7598796Z res = mod(**inputs) 2025-11-03T16:33:19.7599154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7599544Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7599925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 875, in forward 2025-11-03T16:33:19.7600307Z causal_mask = create_causal_mask( 2025-11-03T16:33:19.7600681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 787, in create_causal_mask 2025-11-03T16:33:19.7601173Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-11-03T16:33:19.7601686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 739, in _preprocess_mask_arguments 2025-11-03T16:33:19.7602122Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-11-03T16:33:19.7602602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-11-03T16:33:19.7603013Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-11-03T16:33:19.7603166Z 2025-11-03T16:33:19.7603249Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7603473Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7603804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7604119Z res = mod(**inputs) 2025-11-03T16:33:19.7604462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7604851Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7605227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7605572Z outputs = block( 2025-11-03T16:33:19.7605887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7606232Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7606592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7606936Z return func(*args, **kwargs) 2025-11-03T16:33:19.7607308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7607683Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7608054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7608409Z return func(*args, **kwargs) 2025-11-03T16:33:19.7608763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7609172Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7609611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7610087Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7610259Z 2025-11-03T16:33:19.7610362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7610692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7610998Z res = mod(**inputs) 2025-11-03T16:33:19.7611331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7611703Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7612063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7612414Z outputs = block( 2025-11-03T16:33:19.7612722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7613068Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7613605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7613966Z return func(*args, **kwargs) 2025-11-03T16:33:19.7614318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7614689Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7615057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7615409Z return func(*args, **kwargs) 2025-11-03T16:33:19.7615859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7616229Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7616577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7616987Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7617155Z 2025-11-03T16:33:19.7617255Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7617595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7617902Z res = mod(**inputs) 2025-11-03T16:33:19.7618242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7618618Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7618980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7619328Z outputs = block( 2025-11-03T16:33:19.7619633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7619979Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7620338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7620706Z return func(*args, **kwargs) 2025-11-03T16:33:19.7621053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7621439Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7621826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7622184Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7622523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7622903Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7623089Z 2025-11-03T16:33:19.7623197Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7623540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7623846Z res = mod(**inputs) 2025-11-03T16:33:19.7624187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7624564Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7624936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7625291Z outputs = block( 2025-11-03T16:33:19.7625596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7625942Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7626305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7626662Z return func(*args, **kwargs) 2025-11-03T16:33:19.7627006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7627404Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7627800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7628181Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7628528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7629003Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7629235Z 2025-11-03T16:33:19.7629337Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7629684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7629996Z res = mod(**inputs) 2025-11-03T16:33:19.7630338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7630712Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7631086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7631445Z outputs = block( 2025-11-03T16:33:19.7631760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7632102Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7632473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7632854Z return func(*args, **kwargs) 2025-11-03T16:33:19.7633232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7633660Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7634154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7634569Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7634939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7635347Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7635530Z 2025-11-03T16:33:19.7635645Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7635998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7636321Z res = mod(**inputs) 2025-11-03T16:33:19.7636701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7637099Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7637485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7637877Z outputs = block( 2025-11-03T16:33:19.7638216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7638592Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7638982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7639365Z return func(*args, **kwargs) 2025-11-03T16:33:19.7639742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7640154Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7640551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7640935Z return func(*args, **kwargs) 2025-11-03T16:33:19.7641304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7641810Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7642240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7642612Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7642771Z 2025-11-03T16:33:19.7642888Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7643107Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7643438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7643736Z res = mod(**inputs) 2025-11-03T16:33:19.7644070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7644436Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7644798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7645144Z outputs = block( 2025-11-03T16:33:19.7645451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7645790Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7646139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7646488Z return func(*args, **kwargs) 2025-11-03T16:33:19.7646840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7647208Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7647562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7647923Z return func(*args, **kwargs) 2025-11-03T16:33:19.7648267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7648650Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7649068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7649513Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7649693Z 2025-11-03T16:33:19.7649789Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7650148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7650450Z res = mod(**inputs) 2025-11-03T16:33:19.7650785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7651151Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7651514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7651863Z outputs = block( 2025-11-03T16:33:19.7652164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7652500Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7652856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7653207Z return func(*args, **kwargs) 2025-11-03T16:33:19.7653551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7653921Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7654277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7654626Z return func(*args, **kwargs) 2025-11-03T16:33:19.7654967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7655333Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7655701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7656065Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7656231Z 2025-11-03T16:33:19.7656328Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7656659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7656961Z res = mod(**inputs) 2025-11-03T16:33:19.7657284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7657653Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7658013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7658360Z outputs = block( 2025-11-03T16:33:19.7658662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7658996Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7659350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7659698Z return func(*args, **kwargs) 2025-11-03T16:33:19.7660042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7660419Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7660817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7661179Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7661516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7661888Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7662047Z 2025-11-03T16:33:19.7662147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7662482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7662809Z res = mod(**inputs) 2025-11-03T16:33:19.7663143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7663512Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7663869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7664218Z outputs = block( 2025-11-03T16:33:19.7664525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7664864Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7665214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7665564Z return func(*args, **kwargs) 2025-11-03T16:33:19.7665907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7666293Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7666679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7667036Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7667368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7667792Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7668007Z 2025-11-03T16:33:19.7668113Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7668479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7668782Z res = mod(**inputs) 2025-11-03T16:33:19.7669124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7669499Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7669867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7670220Z outputs = block( 2025-11-03T16:33:19.7670519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7670864Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7671225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7671577Z return func(*args, **kwargs) 2025-11-03T16:33:19.7671920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7672306Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7672698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7673093Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7673459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7673835Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7674082Z 2025-11-03T16:33:19.7674186Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7674544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7674877Z res = mod(**inputs) 2025-11-03T16:33:19.7675236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7675636Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7676048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7676424Z outputs = block( 2025-11-03T16:33:19.7676749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7677105Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7677491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7677863Z return func(*args, **kwargs) 2025-11-03T16:33:19.7678228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7678622Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7679010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7679382Z return func(*args, **kwargs) 2025-11-03T16:33:19.7679749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7680234Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7680693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7681083Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7681262Z 2025-11-03T16:33:19.7681344Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7681586Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7682008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7682327Z res = mod(**inputs) 2025-11-03T16:33:19.7682690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7683082Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7683470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7683848Z outputs = block( 2025-11-03T16:33:19.7684152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7684504Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7684865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7685225Z return func(*args, **kwargs) 2025-11-03T16:33:19.7685577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7685956Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7686331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7686688Z return func(*args, **kwargs) 2025-11-03T16:33:19.7687041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7687441Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7687869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7688331Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7688505Z 2025-11-03T16:33:19.7688613Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7688961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7689265Z res = mod(**inputs) 2025-11-03T16:33:19.7689625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7690002Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7690374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7690732Z outputs = block( 2025-11-03T16:33:19.7691036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7691384Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7691756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7692102Z return func(*args, **kwargs) 2025-11-03T16:33:19.7692442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7692813Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7693178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7693526Z return func(*args, **kwargs) 2025-11-03T16:33:19.7693871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7694238Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7694576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7694950Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7695113Z 2025-11-03T16:33:19.7695219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7695588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7695890Z res = mod(**inputs) 2025-11-03T16:33:19.7696223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7696595Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7696957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7697299Z outputs = block( 2025-11-03T16:33:19.7697605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7697948Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7698310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7698661Z return func(*args, **kwargs) 2025-11-03T16:33:19.7699004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7699393Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7699773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7700139Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7700482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7700851Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7701018Z 2025-11-03T16:33:19.7701115Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7701452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7701753Z res = mod(**inputs) 2025-11-03T16:33:19.7702082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7702478Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7702849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7703199Z outputs = block( 2025-11-03T16:33:19.7703509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7703844Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7704200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7704550Z return func(*args, **kwargs) 2025-11-03T16:33:19.7704897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7705278Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7705664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7706029Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7706362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7706788Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7707005Z 2025-11-03T16:33:19.7707103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7707443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7707745Z res = mod(**inputs) 2025-11-03T16:33:19.7708080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7708496Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7708861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7709220Z outputs = block( 2025-11-03T16:33:19.7709536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7709886Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7710243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7710602Z return func(*args, **kwargs) 2025-11-03T16:33:19.7710957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7711354Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7711748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7712119Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7712474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7712861Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7713028Z 2025-11-03T16:33:19.7713162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7713691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7714047Z res = mod(**inputs) 2025-11-03T16:33:19.7714420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7714827Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7715286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7715653Z outputs = block( 2025-11-03T16:33:19.7715961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7716366Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7716735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7717104Z return func(*args, **kwargs) 2025-11-03T16:33:19.7717454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:33:19.7717855Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:33:19.7718020Z 2025-11-03T16:33:19.7718121Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7718471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7718788Z res = mod(**inputs) 2025-11-03T16:33:19.7719130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7719515Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7719891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7720253Z outputs = block( 2025-11-03T16:33:19.7720559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7720944Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7721308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7721671Z return func(*args, **kwargs) 2025-11-03T16:33:19.7722073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7722460Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7722854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7723234Z return func(*args, **kwargs) 2025-11-03T16:33:19.7723601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7724096Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7724552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7724948Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7725127Z 2025-11-03T16:33:19.7725209Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7725457Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7725814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7726140Z res = mod(**inputs) 2025-11-03T16:33:19.7726497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7726898Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7727308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7727669Z outputs = block( 2025-11-03T16:33:19.7727992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7728351Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7728728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7729102Z return func(*args, **kwargs) 2025-11-03T16:33:19.7729461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7729851Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7730217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7730573Z return func(*args, **kwargs) 2025-11-03T16:33:19.7730915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7731304Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7731732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7732189Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7732361Z 2025-11-03T16:33:19.7732470Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7732799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7733107Z res = mod(**inputs) 2025-11-03T16:33:19.7733445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7733819Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7734186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7734529Z outputs = block( 2025-11-03T16:33:19.7734834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7735181Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7735650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7736007Z return func(*args, **kwargs) 2025-11-03T16:33:19.7736379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7736825Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7737193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7737546Z return func(*args, **kwargs) 2025-11-03T16:33:19.7737885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7738253Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7738588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7738966Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7739131Z 2025-11-03T16:33:19.7739236Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7739567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7739870Z res = mod(**inputs) 2025-11-03T16:33:19.7740206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7740588Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7740952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7741308Z outputs = block( 2025-11-03T16:33:19.7741619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7741977Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7742335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7742678Z return func(*args, **kwargs) 2025-11-03T16:33:19.7743036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7743447Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7743842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7744220Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7744561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7744942Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7745112Z 2025-11-03T16:33:19.7745210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7745560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7745863Z res = mod(**inputs) 2025-11-03T16:33:19.7746206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7746591Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7746968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7747331Z outputs = block( 2025-11-03T16:33:19.7747639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7747996Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7748372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7748732Z return func(*args, **kwargs) 2025-11-03T16:33:19.7749114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7749500Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7749894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7750267Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7750607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7751046Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7751279Z 2025-11-03T16:33:19.7751382Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7751736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7752059Z res = mod(**inputs) 2025-11-03T16:33:19.7752413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7752791Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7753175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7753545Z outputs = block( 2025-11-03T16:33:19.7753862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7754315Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7754688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7755061Z return func(*args, **kwargs) 2025-11-03T16:33:19.7755436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7755861Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7756252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7756652Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7757002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7757552Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7757714Z 2025-11-03T16:33:19.7757820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7758158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7758471Z res = mod(**inputs) 2025-11-03T16:33:19.7758812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7759192Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7759568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7759920Z outputs = block( 2025-11-03T16:33:19.7760228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7760574Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7760940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7761297Z return func(*args, **kwargs) 2025-11-03T16:33:19.7761641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7762020Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7762442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7762803Z return func(*args, **kwargs) 2025-11-03T16:33:19.7763149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7763624Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7764071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7764456Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7764620Z 2025-11-03T16:33:19.7764709Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7764934Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7765280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7765591Z res = mod(**inputs) 2025-11-03T16:33:19.7765935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7766316Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7766683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7767041Z outputs = block( 2025-11-03T16:33:19.7767356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7767722Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7768087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7768453Z return func(*args, **kwargs) 2025-11-03T16:33:19.7768816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7769210Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7769595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7770004Z return func(*args, **kwargs) 2025-11-03T16:33:19.7770361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7770752Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7771191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7771672Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7771852Z 2025-11-03T16:33:19.7771955Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7772311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7772635Z res = mod(**inputs) 2025-11-03T16:33:19.7772994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7773386Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7773773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7774141Z outputs = block( 2025-11-03T16:33:19.7774463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7774823Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7775188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7775557Z return func(*args, **kwargs) 2025-11-03T16:33:19.7775923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7776348Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7776724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7777091Z return func(*args, **kwargs) 2025-11-03T16:33:19.7777453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7777843Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7778201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7778587Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7778762Z 2025-11-03T16:33:19.7778864Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7779220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7779542Z res = mod(**inputs) 2025-11-03T16:33:19.7779894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7780278Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7780663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7781032Z outputs = block( 2025-11-03T16:33:19.7781369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7781722Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7782096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7782461Z return func(*args, **kwargs) 2025-11-03T16:33:19.7782843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7783249Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7783700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7784107Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7784462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7784855Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7785024Z 2025-11-03T16:33:19.7785134Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7785479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7785794Z res = mod(**inputs) 2025-11-03T16:33:19.7786142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7786533Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7786909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7787273Z outputs = block( 2025-11-03T16:33:19.7787602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7787942Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7788296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7788639Z return func(*args, **kwargs) 2025-11-03T16:33:19.7788983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7789366Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7789783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7790149Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7790473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7790896Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7791123Z 2025-11-03T16:33:19.7791219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7791555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7791853Z res = mod(**inputs) 2025-11-03T16:33:19.7792179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7792547Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7792915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7793263Z outputs = block( 2025-11-03T16:33:19.7793560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7793899Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7794342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7794738Z return func(*args, **kwargs) 2025-11-03T16:33:19.7795103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7795506Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7795892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7796265Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7796605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7796993Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7797154Z 2025-11-03T16:33:19.7797254Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7797593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7797901Z res = mod(**inputs) 2025-11-03T16:33:19.7798236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7798597Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7798960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7799308Z outputs = block( 2025-11-03T16:33:19.7799612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7799953Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7800304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7800659Z return func(*args, **kwargs) 2025-11-03T16:33:19.7801003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:33:19.7801395Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:33:19.7801544Z 2025-11-03T16:33:19.7801648Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7801973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7802276Z res = mod(**inputs) 2025-11-03T16:33:19.7802640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7803014Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7803376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7803729Z outputs = block( 2025-11-03T16:33:19.7804041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7804383Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7804739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7805084Z return func(*args, **kwargs) 2025-11-03T16:33:19.7805432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7805804Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7806170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7806515Z return func(*args, **kwargs) 2025-11-03T16:33:19.7806864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7807324Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7807782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7808152Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7808312Z 2025-11-03T16:33:19.7808389Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7808613Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7808947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7809249Z res = mod(**inputs) 2025-11-03T16:33:19.7809587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7809982Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7810361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7810712Z outputs = block( 2025-11-03T16:33:19.7811022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7811396Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7811791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7812154Z return func(*args, **kwargs) 2025-11-03T16:33:19.7812516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7812898Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7813379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7813748Z return func(*args, **kwargs) 2025-11-03T16:33:19.7814115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7814517Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7814958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7815426Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7815613Z 2025-11-03T16:33:19.7815716Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7816152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7816515Z res = mod(**inputs) 2025-11-03T16:33:19.7816852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7817232Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7817610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7817974Z outputs = block( 2025-11-03T16:33:19.7818287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7818629Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7818997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7819356Z return func(*args, **kwargs) 2025-11-03T16:33:19.7819715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7820096Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7820464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7820819Z return func(*args, **kwargs) 2025-11-03T16:33:19.7821200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7821570Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7821911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7822296Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7822466Z 2025-11-03T16:33:19.7822565Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7822914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7823223Z res = mod(**inputs) 2025-11-03T16:33:19.7823587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7823966Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7824344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7824710Z outputs = block( 2025-11-03T16:33:19.7825025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7825370Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7825737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7826093Z return func(*args, **kwargs) 2025-11-03T16:33:19.7826451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7826848Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7827243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7827624Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7827963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7828331Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7828490Z 2025-11-03T16:33:19.7828589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7828928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7829231Z res = mod(**inputs) 2025-11-03T16:33:19.7829603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7829985Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7830350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7830702Z outputs = block( 2025-11-03T16:33:19.7831015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7831363Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7831714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7832072Z return func(*args, **kwargs) 2025-11-03T16:33:19.7832424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7832823Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7833211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7833576Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7833963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7834439Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7834667Z 2025-11-03T16:33:19.7834782Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7835143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7835453Z res = mod(**inputs) 2025-11-03T16:33:19.7835804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7836199Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7836570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7836924Z outputs = block( 2025-11-03T16:33:19.7837226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7837564Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7837921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7838268Z return func(*args, **kwargs) 2025-11-03T16:33:19.7838607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7838992Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7839370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7839735Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7840072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7840430Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7840596Z 2025-11-03T16:33:19.7840699Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7841028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7841327Z res = mod(**inputs) 2025-11-03T16:33:19.7841648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7842014Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7842409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7842767Z outputs = block( 2025-11-03T16:33:19.7843067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7843403Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7843763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7844112Z return func(*args, **kwargs) 2025-11-03T16:33:19.7844460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7844831Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7845186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7845538Z return func(*args, **kwargs) 2025-11-03T16:33:19.7845896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7846375Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7846817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7847203Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7847393Z 2025-11-03T16:33:19.7847473Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7847703Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7848046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7848349Z res = mod(**inputs) 2025-11-03T16:33:19.7848693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7849075Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7849444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7849814Z outputs = block( 2025-11-03T16:33:19.7850120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7850472Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7850838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7851198Z return func(*args, **kwargs) 2025-11-03T16:33:19.7851551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7851932Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7852313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7852670Z return func(*args, **kwargs) 2025-11-03T16:33:19.7853024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7853407Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7853835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7854297Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7854470Z 2025-11-03T16:33:19.7854577Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7854920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7855225Z res = mod(**inputs) 2025-11-03T16:33:19.7855597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7855974Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7856345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7856698Z outputs = block( 2025-11-03T16:33:19.7857013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7857364Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7857731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7858090Z return func(*args, **kwargs) 2025-11-03T16:33:19.7858439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7858818Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7859194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7859554Z return func(*args, **kwargs) 2025-11-03T16:33:19.7859910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7860276Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7860622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7861024Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7861188Z 2025-11-03T16:33:19.7861295Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7861630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7861945Z res = mod(**inputs) 2025-11-03T16:33:19.7862288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7862668Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7863045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7863427Z outputs = block( 2025-11-03T16:33:19.7863742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7864092Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7864456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7864814Z return func(*args, **kwargs) 2025-11-03T16:33:19.7865163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7865556Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7865950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7866381Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7866723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7867110Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7867283Z 2025-11-03T16:33:19.7867382Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7867726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7868038Z res = mod(**inputs) 2025-11-03T16:33:19.7868380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7868765Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7869180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7869545Z outputs = block( 2025-11-03T16:33:19.7869851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7870199Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7870561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7870920Z return func(*args, **kwargs) 2025-11-03T16:33:19.7871271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7871660Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7872050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7872427Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7872766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7873202Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7873417Z 2025-11-03T16:33:19.7873517Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7873893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7874272Z res = mod(**inputs) 2025-11-03T16:33:19.7874631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7875021Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7875398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7875768Z outputs = block( 2025-11-03T16:33:19.7876089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7876470Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7876832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7877203Z return func(*args, **kwargs) 2025-11-03T16:33:19.7877570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7877974Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7878376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7878759Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7879121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7879511Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7879680Z 2025-11-03T16:33:19.7879802Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7880156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7880465Z res = mod(**inputs) 2025-11-03T16:33:19.7880817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7881203Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7881588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7881946Z outputs = block( 2025-11-03T16:33:19.7882267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7882651Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7883034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7883408Z return func(*args, **kwargs) 2025-11-03T16:33:19.7883761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:33:19.7884173Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:33:19.7884334Z 2025-11-03T16:33:19.7884437Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7884788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7885107Z res = mod(**inputs) 2025-11-03T16:33:19.7885451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7885842Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7886232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7886608Z outputs = block( 2025-11-03T16:33:19.7886917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7887275Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7887667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7888035Z return func(*args, **kwargs) 2025-11-03T16:33:19.7888395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7888778Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7889143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7889493Z return func(*args, **kwargs) 2025-11-03T16:33:19.7889838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7890310Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7890736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7891109Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7891276Z 2025-11-03T16:33:19.7891351Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7891578Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7891907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7892207Z res = mod(**inputs) 2025-11-03T16:33:19.7892549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7892925Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7893292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7893632Z outputs = block( 2025-11-03T16:33:19.7893934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7894271Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7894625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7894965Z return func(*args, **kwargs) 2025-11-03T16:33:19.7895313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7895682Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7896083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7896433Z return func(*args, **kwargs) 2025-11-03T16:33:19.7896767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7897144Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7897562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7898010Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7898179Z 2025-11-03T16:33:19.7898282Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7898611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7898915Z res = mod(**inputs) 2025-11-03T16:33:19.7899254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7899626Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7899986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7900325Z outputs = block( 2025-11-03T16:33:19.7900651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7900991Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7901349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7901689Z return func(*args, **kwargs) 2025-11-03T16:33:19.7902038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7902413Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7902780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7903148Z return func(*args, **kwargs) 2025-11-03T16:33:19.7903483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7903850Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7904184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7904554Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7904715Z 2025-11-03T16:33:19.7904820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7905145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7905449Z res = mod(**inputs) 2025-11-03T16:33:19.7905782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7906149Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7906501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7906847Z outputs = block( 2025-11-03T16:33:19.7907150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7907492Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7907843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7908244Z return func(*args, **kwargs) 2025-11-03T16:33:19.7908629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7909029Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7909419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7909785Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7910126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7910507Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7910671Z 2025-11-03T16:33:19.7910776Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7911115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7911418Z res = mod(**inputs) 2025-11-03T16:33:19.7911757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7912138Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7912509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7912861Z outputs = block( 2025-11-03T16:33:19.7913160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7913659Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7914083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7914446Z return func(*args, **kwargs) 2025-11-03T16:33:19.7914796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7915203Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7915613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7915982Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7916363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7916797Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7917028Z 2025-11-03T16:33:19.7917128Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7917479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7917794Z res = mod(**inputs) 2025-11-03T16:33:19.7918140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7918515Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7918894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7919256Z outputs = block( 2025-11-03T16:33:19.7919571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7919926Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7920287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7920648Z return func(*args, **kwargs) 2025-11-03T16:33:19.7921007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7921406Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7921790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7922219Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7922574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7922969Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7923136Z 2025-11-03T16:33:19.7923243Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7923587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7923903Z res = mod(**inputs) 2025-11-03T16:33:19.7924254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7924639Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7925015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7925374Z outputs = block( 2025-11-03T16:33:19.7925697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7926053Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7926429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7926796Z return func(*args, **kwargs) 2025-11-03T16:33:19.7927163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7927571Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7927951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7928308Z return func(*args, **kwargs) 2025-11-03T16:33:19.7928658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7929135Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7929573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7929976Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7930136Z 2025-11-03T16:33:19.7930220Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7930440Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7930774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7931078Z res = mod(**inputs) 2025-11-03T16:33:19.7931412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7931772Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7932137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7932486Z outputs = block( 2025-11-03T16:33:19.7932790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7933129Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7933474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7933826Z return func(*args, **kwargs) 2025-11-03T16:33:19.7934169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7934541Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7934897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7935244Z return func(*args, **kwargs) 2025-11-03T16:33:19.7935616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7936000Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7936426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7936872Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7937048Z 2025-11-03T16:33:19.7937145Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7937483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7937785Z res = mod(**inputs) 2025-11-03T16:33:19.7938123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7938494Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7938861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7939214Z outputs = block( 2025-11-03T16:33:19.7939523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7939860Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7940219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7940582Z return func(*args, **kwargs) 2025-11-03T16:33:19.7940925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7941290Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7941649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7941994Z return func(*args, **kwargs) 2025-11-03T16:33:19.7942338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7942428Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7942643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7942753Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7942758Z 2025-11-03T16:33:19.7942864Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7943051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7943112Z res = mod(**inputs) 2025-11-03T16:33:19.7943354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7943432Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7943673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7943735Z outputs = block( 2025-11-03T16:33:19.7943952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7944028Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7944252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7944335Z return func(*args, **kwargs) 2025-11-03T16:33:19.7944564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7944667Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7944897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7945039Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7945254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7945365Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7945368Z 2025-11-03T16:33:19.7945473Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7945659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7945726Z res = mod(**inputs) 2025-11-03T16:33:19.7945963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7946041Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7946281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7946344Z outputs = block( 2025-11-03T16:33:19.7946560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7946636Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7946868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7946943Z return func(*args, **kwargs) 2025-11-03T16:33:19.7947187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7947292Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7947520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7947595Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7947811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7947985Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7948003Z 2025-11-03T16:33:19.7948112Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7948302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7948374Z res = mod(**inputs) 2025-11-03T16:33:19.7948616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7948694Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7948940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7949001Z outputs = block( 2025-11-03T16:33:19.7949221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7949298Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7949530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7949605Z return func(*args, **kwargs) 2025-11-03T16:33:19.7949840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7949947Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7950184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7950272Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7950482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7950592Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7950635Z 2025-11-03T16:33:19.7950742Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7950932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7951003Z res = mod(**inputs) 2025-11-03T16:33:19.7951245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7951324Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7951568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7951627Z outputs = block( 2025-11-03T16:33:19.7951847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7951922Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7952156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7952230Z return func(*args, **kwargs) 2025-11-03T16:33:19.7952467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:33:19.7952575Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:33:19.7952578Z 2025-11-03T16:33:19.7952674Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7952885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7952945Z res = mod(**inputs) 2025-11-03T16:33:19.7953184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7953271Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7953508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7953575Z outputs = block( 2025-11-03T16:33:19.7953789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7953880Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7954190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7954268Z return func(*args, **kwargs) 2025-11-03T16:33:19.7954520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7954607Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7954852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7954931Z return func(*args, **kwargs) 2025-11-03T16:33:19.7955167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7955354Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7955567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7955686Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7955691Z 2025-11-03T16:33:19.7955774Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7955874Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7956075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7956136Z res = mod(**inputs) 2025-11-03T16:33:19.7956385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7956496Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7956745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7956808Z outputs = block( 2025-11-03T16:33:19.7957020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7957103Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7957337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7957413Z return func(*args, **kwargs) 2025-11-03T16:33:19.7957648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7957734Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7957976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7958044Z return func(*args, **kwargs) 2025-11-03T16:33:19.7958285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7958381Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7958661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7958810Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7958813Z 2025-11-03T16:33:19.7958912Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7959108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7959169Z res = mod(**inputs) 2025-11-03T16:33:19.7959423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7959502Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7959741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7959823Z outputs = block( 2025-11-03T16:33:19.7960035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7960118Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7960350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7960415Z return func(*args, **kwargs) 2025-11-03T16:33:19.7960656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7960739Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7960977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7961042Z return func(*args, **kwargs) 2025-11-03T16:33:19.7961277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7961363Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7961571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7961691Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7961694Z 2025-11-03T16:33:19.7961792Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7961985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7962046Z res = mod(**inputs) 2025-11-03T16:33:19.7962314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7962401Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7962641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7962711Z outputs = block( 2025-11-03T16:33:19.7962925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7963003Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7963241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7963307Z return func(*args, **kwargs) 2025-11-03T16:33:19.7963550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7963649Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7963896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7963974Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7964188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7964304Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7964323Z 2025-11-03T16:33:19.7964421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7964617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7964677Z res = mod(**inputs) 2025-11-03T16:33:19.7964917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7965002Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7965242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7965309Z outputs = block( 2025-11-03T16:33:19.7965537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7965614Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7965856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7965924Z return func(*args, **kwargs) 2025-11-03T16:33:19.7966168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7966267Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7966513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7966593Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7966804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7966988Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7966992Z 2025-11-03T16:33:19.7967093Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7967292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7967355Z res = mod(**inputs) 2025-11-03T16:33:19.7967601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7967688Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7967927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7967994Z outputs = block( 2025-11-03T16:33:19.7968247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7968336Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7968575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7968645Z return func(*args, **kwargs) 2025-11-03T16:33:19.7968897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7968997Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7969258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7969343Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7969560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7969681Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7969684Z 2025-11-03T16:33:19.7969789Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7969989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7970051Z res = mod(**inputs) 2025-11-03T16:33:19.7970326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7970407Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7970650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7970721Z outputs = block( 2025-11-03T16:33:19.7970940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7971029Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7971265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7971352Z return func(*args, **kwargs) 2025-11-03T16:33:19.7971602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7971688Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7971935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7972004Z return func(*args, **kwargs) 2025-11-03T16:33:19.7972246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7972438Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7972662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7972787Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7972792Z 2025-11-03T16:33:19.7972874Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7972984Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7973177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7973240Z res = mod(**inputs) 2025-11-03T16:33:19.7973496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7973577Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7973828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7973889Z outputs = block( 2025-11-03T16:33:19.7974150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7974236Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7974472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7974547Z return func(*args, **kwargs) 2025-11-03T16:33:19.7974788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7974881Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7975115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7975183Z return func(*args, **kwargs) 2025-11-03T16:33:19.7975428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7975525Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7975822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7975951Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7975955Z 2025-11-03T16:33:19.7976053Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7976276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7976337Z res = mod(**inputs) 2025-11-03T16:33:19.7976590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7976670Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7976921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7976985Z outputs = block( 2025-11-03T16:33:19.7977202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7977305Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7977543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7977619Z return func(*args, **kwargs) 2025-11-03T16:33:19.7977866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7977954Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7978199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7978269Z return func(*args, **kwargs) 2025-11-03T16:33:19.7978521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7978602Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7978817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7978940Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7978944Z 2025-11-03T16:33:19.7979046Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7979248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7979312Z res = mod(**inputs) 2025-11-03T16:33:19.7979569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7979653Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7979921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7979989Z outputs = block( 2025-11-03T16:33:19.7980206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7980291Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7980524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7980592Z return func(*args, **kwargs) 2025-11-03T16:33:19.7980841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7980941Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7981187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.7981263Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.7981478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7981599Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7981603Z 2025-11-03T16:33:19.7981703Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7981905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7981986Z res = mod(**inputs) 2025-11-03T16:33:19.7982240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7982324Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7982566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7982634Z outputs = block( 2025-11-03T16:33:19.7982852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7982953Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7983193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7983291Z return func(*args, **kwargs) 2025-11-03T16:33:19.7983542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7983644Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7983895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.7983972Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.7984187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.7984367Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.7984371Z 2025-11-03T16:33:19.7984472Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7984676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7984741Z res = mod(**inputs) 2025-11-03T16:33:19.7984998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7985080Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7985330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7985399Z outputs = block( 2025-11-03T16:33:19.7985607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7985688Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7986016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7986093Z return func(*args, **kwargs) 2025-11-03T16:33:19.7986327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.7986423Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.7986662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.7986742Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.7986959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7987070Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7987074Z 2025-11-03T16:33:19.7987170Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7987364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7987425Z res = mod(**inputs) 2025-11-03T16:33:19.7987673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7987751Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7987991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7988070Z outputs = block( 2025-11-03T16:33:19.7988289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7988373Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7988609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7988687Z return func(*args, **kwargs) 2025-11-03T16:33:19.7988930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 452, in forward 2025-11-03T16:33:19.7989048Z hidden_states = residual + feed_forward_hidden_states 2025-11-03T16:33:19.7989051Z 2025-11-03T16:33:19.7989157Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7989346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7989415Z res = mod(**inputs) 2025-11-03T16:33:19.7989653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7989730Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7989976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7990035Z outputs = block( 2025-11-03T16:33:19.7990256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7990330Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7990569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7990636Z return func(*args, **kwargs) 2025-11-03T16:33:19.7990870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7990964Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7991198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7991271Z return func(*args, **kwargs) 2025-11-03T16:33:19.7991509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 314, in forward 2025-11-03T16:33:19.7991718Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-11-03T16:33:19.7991939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7992052Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7992055Z 2025-11-03T16:33:19.7992141Z cudagraph partition due to non gpu ops 2025-11-03T16:33:19.7992241Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7992437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7992499Z res = mod(**inputs) 2025-11-03T16:33:19.7992740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7992826Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7993065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7993132Z outputs = block( 2025-11-03T16:33:19.7993342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7993418Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7993656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7993741Z return func(*args, **kwargs) 2025-11-03T16:33:19.7994071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7994162Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7994407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7994490Z return func(*args, **kwargs) 2025-11-03T16:33:19.7994736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 346, in forward 2025-11-03T16:33:19.7994840Z attn_output, attn_weights = attention_interface( 2025-11-03T16:33:19.7995156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:33:19.7995303Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:33:19.7995308Z 2025-11-03T16:33:19.7995404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7995586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7995655Z res = mod(**inputs) 2025-11-03T16:33:19.7995896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7995985Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7996216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7996276Z outputs = block( 2025-11-03T16:33:19.7996495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.7996571Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.7996801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7996868Z return func(*args, **kwargs) 2025-11-03T16:33:19.7997095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 414, in forward 2025-11-03T16:33:19.7997183Z attn_output, self_attn_weights = self.attn( 2025-11-03T16:33:19.7997408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.7998171Z return func(*args, **kwargs) 2025-11-03T16:33:19.7998403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 359, in forward 2025-11-03T16:33:19.7998489Z attn_output = self.c_proj(attn_output) 2025-11-03T16:33:19.7998692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.7998799Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.7998804Z 2025-11-03T16:33:19.7998908Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.7999092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.7999159Z res = mod(**inputs) 2025-11-03T16:33:19.7999397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.7999478Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.7999719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.7999780Z outputs = block( 2025-11-03T16:33:19.7999999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.8000074Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.8000312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.8000395Z return func(*args, **kwargs) 2025-11-03T16:33:19.8000622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.8000727Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.8000963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-11-03T16:33:19.8001049Z hidden_states = self.c_fc(hidden_states) 2025-11-03T16:33:19.8001256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.8001381Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.8001384Z 2025-11-03T16:33:19.8001515Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.8001705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.8001773Z res = mod(**inputs) 2025-11-03T16:33:19.8002005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.8002091Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.8002324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.8002382Z outputs = block( 2025-11-03T16:33:19.8002599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.8002676Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.8002907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.8002971Z return func(*args, **kwargs) 2025-11-03T16:33:19.8003202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.8003306Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.8003534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-11-03T16:33:19.8003614Z hidden_states = self.act(hidden_states) 2025-11-03T16:33:19.8003842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:19.8004009Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:19.8004021Z 2025-11-03T16:33:19.8004118Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.8004302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.8004369Z res = mod(**inputs) 2025-11-03T16:33:19.8004604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1367, in forward 2025-11-03T16:33:19.8004687Z transformer_outputs = self.transformer( 2025-11-03T16:33:19.8004915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 927, in forward 2025-11-03T16:33:19.8004975Z outputs = block( 2025-11-03T16:33:19.8005189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:19.8005262Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:19.8005492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:33:19.8005557Z return func(*args, **kwargs) 2025-11-03T16:33:19.8005784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 450, in forward 2025-11-03T16:33:19.8005907Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-11-03T16:33:19.8006137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 377, in forward 2025-11-03T16:33:19.8006223Z hidden_states = self.c_proj(hidden_states) 2025-11-03T16:33:19.8006428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-11-03T16:33:19.8006543Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-11-03T16:33:19.8006549Z 2025-11-03T16:33:19.8006644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.8006826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.8006911Z res = mod(**inputs) 2025-11-03T16:33:19.8007144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1381, in forward 2025-11-03T16:33:19.8007224Z logits = self.score(hidden_states) 2025-11-03T16:33:19.8007228Z 2025-11-03T16:33:19.8007323Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.8007504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.8007571Z res = mod(**inputs) 2025-11-03T16:33:19.8007803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1424, in forward 2025-11-03T16:33:19.8007947Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-11-03T16:33:19.8007950Z 2025-11-03T16:33:19.8008044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:19.8008233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:19.8008292Z res = mod(**inputs) 2025-11-03T16:33:19.8008524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1424, in forward 2025-11-03T16:33:19.8008664Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-11-03T16:33:19.8008668Z 2025-11-03T16:33:31.5126164Z Compilation time (from dynamo_timed): 17.399119483 2025-11-03T16:33:31.5130112Z pass 2025-11-03T16:33:31.5134380Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:31.5138897Z TIMING: _recursive_pre_grad_passes:0.01224 _recursive_joint_graph_passes:0.30842 _recursive_post_grad_passes:0.0602 async_compile.wait:0.65087 code_gen:8.79273 inductor_compile:10.18731 backend_compile:12.90166 gc:0.00237 entire_frame_compile:17.39912 total_wall_time:17.39912 2025-11-03T16:33:31.5140675Z STATS: call_* op count: 1126 | FakeTensorMode.__torch_dispatch__:7369 | FakeTensor.__torch_dispatch__:4239 | ProxyTorchDispatchMode.__torch_dispatch__:1656 2025-11-03T16:33:31.5141166Z Dynamo produced 2 graphs covering 1126 ops with 0 graph breaks (0 unique) 2025-11-03T16:33:33.7370979Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:33:33.7375418Z import pynvml # type: ignore[import] 2025-11-03T16:33:36.9281873Z 2025-11-03T16:33:37.7500193Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:33:37.7500555Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:33:37.7510173Z cpu eval GoogleFnet 2025-11-03T16:33:38.1236136Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:38.2756132Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:38.4346851Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:43.4931742Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4932459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4932819Z res = mod(**inputs) 2025-11-03T16:33:43.4933374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4933960Z outputs = self.fnet( 2025-11-03T16:33:43.4934472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4934872Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4935322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4936074Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4936474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4936870Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4937261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4937694Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4938090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4938516Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4938880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4939380Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4939541Z 2025-11-03T16:33:43.4939644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4940052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4940360Z res = mod(**inputs) 2025-11-03T16:33:43.4940705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4941128Z outputs = self.fnet( 2025-11-03T16:33:43.4941465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4941900Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4942456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4942851Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4943293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4943646Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4944112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4944504Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4944962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4945341Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4945787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4946214Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4946406Z 2025-11-03T16:33:43.4946508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4946860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4947256Z res = mod(**inputs) 2025-11-03T16:33:43.4947593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4948102Z outputs = self.fnet( 2025-11-03T16:33:43.4948467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4948904Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4949303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4949728Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4950137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4950508Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4950990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4951386Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4951861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4952234Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4952678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4953078Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4953302Z 2025-11-03T16:33:43.4953411Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4953763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4954265Z res = mod(**inputs) 2025-11-03T16:33:43.4954647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4955164Z outputs = self.fnet( 2025-11-03T16:33:43.4955575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4956041Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4956467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4956859Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4957298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4958359Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4958808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4959210Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4959684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4960064Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4960504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4960895Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4961104Z 2025-11-03T16:33:43.4961226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4961572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4961958Z res = mod(**inputs) 2025-11-03T16:33:43.4962296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4962736Z outputs = self.fnet( 2025-11-03T16:33:43.4963081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4963517Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4963910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4964366Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4964719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4965151Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4965531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4966011Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4966403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4966864Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4967284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4967740Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4967888Z 2025-11-03T16:33:43.4968016Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4968407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4968740Z res = mod(**inputs) 2025-11-03T16:33:43.4969168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4969543Z outputs = self.fnet( 2025-11-03T16:33:43.4969958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4970353Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4970783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4971202Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4971610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4971963Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4972432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4972837Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4973349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4973763Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4974184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4974651Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4974805Z 2025-11-03T16:33:43.4974928Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4975342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4975701Z res = mod(**inputs) 2025-11-03T16:33:43.4976070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4976496Z outputs = self.fnet( 2025-11-03T16:33:43.4976891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4977307Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4977757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4978185Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4978629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4979010Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4979476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4979893Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4980377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4980796Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4981243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4981744Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4981907Z 2025-11-03T16:33:43.4982012Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4982427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4982801Z res = mod(**inputs) 2025-11-03T16:33:43.4983221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4983615Z outputs = self.fnet( 2025-11-03T16:33:43.4984067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4984440Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4984901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4985279Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4985725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4986085Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4986536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4986938Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4987408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4987823Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4988242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4988747Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4988902Z 2025-11-03T16:33:43.4989012Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4989408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4989758Z res = mod(**inputs) 2025-11-03T16:33:43.4990121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4990537Z outputs = self.fnet( 2025-11-03T16:33:43.4990898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4991307Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4991701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4992113Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4992523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4992879Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.4993316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.4993717Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.4994263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.4994651Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.4995104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.4995523Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.4995758Z 2025-11-03T16:33:43.4995862Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.4996209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.4996640Z res = mod(**inputs) 2025-11-03T16:33:43.4996980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.4997434Z outputs = self.fnet( 2025-11-03T16:33:43.4997779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.4998232Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.4998602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.4999080Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.4999488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.4999897Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5000354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5000748Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5001249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5001631Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5002084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5002476Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5002688Z 2025-11-03T16:33:43.5002785Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5003119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5003536Z res = mod(**inputs) 2025-11-03T16:33:43.5003864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5004299Z outputs = self.fnet( 2025-11-03T16:33:43.5004630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5005067Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5005431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5005887Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5006220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5006645Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5007007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5007472Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5007853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5008294Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5008679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5009123Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5009265Z 2025-11-03T16:33:43.5009366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5009763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5010066Z res = mod(**inputs) 2025-11-03T16:33:43.5010477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5010830Z outputs = self.fnet( 2025-11-03T16:33:43.5011242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5011675Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5012100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5012476Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5012891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5013397Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5013777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5014219Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5014605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5015039Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5015401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5015863Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5016020Z 2025-11-03T16:33:43.5016120Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5016535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5016843Z res = mod(**inputs) 2025-11-03T16:33:43.5017207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5017590Z outputs = self.fnet( 2025-11-03T16:33:43.5018003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 512, in forward 2025-11-03T16:33:43.5018453Z embedding_output = self.embeddings( 2025-11-03T16:33:43.5018808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 142, in forward 2025-11-03T16:33:43.5019179Z embeddings = self.projection(embeddings) 2025-11-03T16:33:43.5019316Z 2025-11-03T16:33:43.5019395Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5019696Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5020035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5020334Z res = mod(**inputs) 2025-11-03T16:33:43.5020667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5021020Z outputs = self.fnet( 2025-11-03T16:33:43.5021356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5021759Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5022123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5022494Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5022842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5023217Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5023569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5023948Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5024324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5024696Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5025054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5025495Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5025677Z 2025-11-03T16:33:43.5025773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5026107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5026408Z res = mod(**inputs) 2025-11-03T16:33:43.5026733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5027077Z outputs = self.fnet( 2025-11-03T16:33:43.5027410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5027767Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5028119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5028478Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5028836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5029177Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5029534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5029914Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5030290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5030666Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5031036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5031467Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5031622Z 2025-11-03T16:33:43.5031730Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5032071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5032395Z res = mod(**inputs) 2025-11-03T16:33:43.5032786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5033156Z outputs = self.fnet( 2025-11-03T16:33:43.5033490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5033865Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5034356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5034762Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5035128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5035493Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5035868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5036270Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5036692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5037064Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5037424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5037814Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5037968Z 2025-11-03T16:33:43.5038069Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5038409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5038737Z res = mod(**inputs) 2025-11-03T16:33:43.5039080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5039445Z outputs = self.fnet( 2025-11-03T16:33:43.5039792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5040162Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5040518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5040897Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5041252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5041606Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5041978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5042361Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5042748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5043120Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5043485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5043874Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5044028Z 2025-11-03T16:33:43.5044126Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5044470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5044812Z res = mod(**inputs) 2025-11-03T16:33:43.5045150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5045505Z outputs = self.fnet( 2025-11-03T16:33:43.5045902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5046327Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5046680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5047049Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5047387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5047728Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5048087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5048459Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5048832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5049208Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5049596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5050068Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5050466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5050834Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5050971Z 2025-11-03T16:33:43.5051071Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5051412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5051718Z res = mod(**inputs) 2025-11-03T16:33:43.5052283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5052648Z outputs = self.fnet( 2025-11-03T16:33:43.5052999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5053363Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5053720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5054095Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5054438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5054779Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5055142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5055511Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5055883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5056259Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5056645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5057072Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5057467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5057852Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5058247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5058682Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5058900Z 2025-11-03T16:33:43.5059008Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5059342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5059639Z res = mod(**inputs) 2025-11-03T16:33:43.5059968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5060318Z outputs = self.fnet( 2025-11-03T16:33:43.5060653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5061007Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5061353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5061719Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5062063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5062404Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5062755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5063148Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5063527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5063904Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5064291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5064724Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5065137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5065523Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5065654Z 2025-11-03T16:33:43.5065742Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5065972Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5066305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5066613Z res = mod(**inputs) 2025-11-03T16:33:43.5066945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5067301Z outputs = self.fnet( 2025-11-03T16:33:43.5067628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5067986Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5068340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5068716Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5069061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5069397Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5069759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5070145Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5070521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5070885Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5071271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5071657Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5071814Z 2025-11-03T16:33:43.5071912Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5072367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5072681Z res = mod(**inputs) 2025-11-03T16:33:43.5073024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5073388Z outputs = self.fnet( 2025-11-03T16:33:43.5073732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5074173Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5074539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5074929Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5075292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5075650Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5076022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5076467Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5076864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5077244Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5077619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5078013Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5078179Z 2025-11-03T16:33:43.5078279Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5078628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5078965Z res = mod(**inputs) 2025-11-03T16:33:43.5079310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5079683Z outputs = self.fnet( 2025-11-03T16:33:43.5080033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5080408Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5080780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5081159Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5081524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5081882Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5082262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5082664Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5083053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5083444Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5083823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5084229Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5084379Z 2025-11-03T16:33:43.5084483Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5084855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5085169Z res = mod(**inputs) 2025-11-03T16:33:43.5085507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5085870Z outputs = self.fnet( 2025-11-03T16:33:43.5086206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5086578Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5086935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5087312Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5087662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5087999Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5088370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5088761Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5089146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5089515Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5089895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5090290Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5090447Z 2025-11-03T16:33:43.5090546Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5090892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5091203Z res = mod(**inputs) 2025-11-03T16:33:43.5091540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5091904Z outputs = self.fnet( 2025-11-03T16:33:43.5092261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5092628Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5092977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5093357Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5093697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5094032Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5094389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5094749Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5095127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5095502Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5095885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5096310Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5096697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5097062Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5097197Z 2025-11-03T16:33:43.5097295Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5097630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5097966Z res = mod(**inputs) 2025-11-03T16:33:43.5098294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5098656Z outputs = self.fnet( 2025-11-03T16:33:43.5098992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5099360Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5099713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5100089Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5100443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5100793Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5101154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5101549Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5101929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5102317Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5102710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5103138Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5103532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5103916Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5104274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5104703Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5104917Z 2025-11-03T16:33:43.5105013Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5105361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5105668Z res = mod(**inputs) 2025-11-03T16:33:43.5106001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5106357Z outputs = self.fnet( 2025-11-03T16:33:43.5106682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5107040Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5107392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5107763Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5108108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5108451Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5108807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5109175Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5109559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5109930Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5110320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5110761Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5111214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5111594Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5111728Z 2025-11-03T16:33:43.5111808Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5112045Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5112395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5112714Z res = mod(**inputs) 2025-11-03T16:33:43.5113053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5113543Z outputs = self.fnet( 2025-11-03T16:33:43.5113890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5114309Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5114677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5115049Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5115409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5115762Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5116134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5116574Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5116958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5117338Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5117707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5118109Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5118260Z 2025-11-03T16:33:43.5118368Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5118730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5119045Z res = mod(**inputs) 2025-11-03T16:33:43.5119386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5119750Z outputs = self.fnet( 2025-11-03T16:33:43.5120085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5120454Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5120813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5121192Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5121548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5121889Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5122262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5122656Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5123043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5123419Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5123781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5124296Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5124454Z 2025-11-03T16:33:43.5124565Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5125007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5125321Z res = mod(**inputs) 2025-11-03T16:33:43.5125649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5125999Z outputs = self.fnet( 2025-11-03T16:33:43.5126330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5140819Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5141392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5141795Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5142161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5142523Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5142939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5143390Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5143791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5144262Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5144635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5145026Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5145179Z 2025-11-03T16:33:43.5145293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5145641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5145954Z res = mod(**inputs) 2025-11-03T16:33:43.5146295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5146693Z outputs = self.fnet( 2025-11-03T16:33:43.5147029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5147393Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5147746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5148118Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5148462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5148803Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5149156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5149535Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5149909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5150300Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5150671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5151059Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5151217Z 2025-11-03T16:33:43.5151319Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5151667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5151980Z res = mod(**inputs) 2025-11-03T16:33:43.5152321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5152728Z outputs = self.fnet( 2025-11-03T16:33:43.5153076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5153447Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5153815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5154322Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5154696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5155063Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5155460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5155844Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5156237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5156630Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5157035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5157484Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5157900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5158294Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5158435Z 2025-11-03T16:33:43.5158536Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5158885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5159231Z res = mod(**inputs) 2025-11-03T16:33:43.5159564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5159925Z outputs = self.fnet( 2025-11-03T16:33:43.5160264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5160659Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5161030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5161413Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5161776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5162133Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5162507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5162891Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5163283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5163674Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5164074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5164516Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5164925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5165334Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5165710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5166153Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5166434Z 2025-11-03T16:33:43.5166544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5166876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5167186Z res = mod(**inputs) 2025-11-03T16:33:43.5167528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5167891Z outputs = self.fnet( 2025-11-03T16:33:43.5168233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5168606Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5168963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5169344Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5169699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5170049Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5170417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5170794Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5171193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5171598Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5171992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5172447Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5172875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5173265Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5173402Z 2025-11-03T16:33:43.5173491Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5173741Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5174094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5174410Z res = mod(**inputs) 2025-11-03T16:33:43.5174761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5175131Z outputs = self.fnet( 2025-11-03T16:33:43.5175472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5175847Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5176214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5176607Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5176959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5177316Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5177695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5178095Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5178495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5178872Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5179252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5179659Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5179814Z 2025-11-03T16:33:43.5179953Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5180312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5180632Z res = mod(**inputs) 2025-11-03T16:33:43.5180985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5181363Z outputs = self.fnet( 2025-11-03T16:33:43.5181719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5182094Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5182466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5182879Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5183249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5183606Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5183978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5184380Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5184780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5185179Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5185554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5185949Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5186109Z 2025-11-03T16:33:43.5186210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5186565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5186882Z res = mod(**inputs) 2025-11-03T16:33:43.5187230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5187619Z outputs = self.fnet( 2025-11-03T16:33:43.5187967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5188354Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5188708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5189068Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5189410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5189746Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5190107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5190485Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5190851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5191211Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5191569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5191948Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5192091Z 2025-11-03T16:33:43.5192186Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5192522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5192825Z res = mod(**inputs) 2025-11-03T16:33:43.5193216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5193574Z outputs = self.fnet( 2025-11-03T16:33:43.5193898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5194343Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5194711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5195105Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5195449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5195783Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5196145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5196532Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5196908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5197263Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5197625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5198007Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5198166Z 2025-11-03T16:33:43.5198272Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5198616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5198914Z res = mod(**inputs) 2025-11-03T16:33:43.5199256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5199613Z outputs = self.fnet( 2025-11-03T16:33:43.5199953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5200310Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5200676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5201045Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5201386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5201719Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5202077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5202446Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5202823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5203192Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5203567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5203986Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5204377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5204742Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5204868Z 2025-11-03T16:33:43.5204971Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5205297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5205603Z res = mod(**inputs) 2025-11-03T16:33:43.5205929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5206313Z outputs = self.fnet( 2025-11-03T16:33:43.5206635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5206993Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5207344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5207714Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5208055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5208385Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5208743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5209113Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5209533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5210068Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5210457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5210877Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5211271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5211683Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5212034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5212463Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5212689Z 2025-11-03T16:33:43.5212787Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5213123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5213617Z res = mod(**inputs) 2025-11-03T16:33:43.5214020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5214367Z outputs = self.fnet( 2025-11-03T16:33:43.5214697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5215056Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5215407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5215767Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5216109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5216449Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5216812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5217195Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5217561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5217934Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5218314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5218745Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5219150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5219514Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5219648Z 2025-11-03T16:33:43.5219780Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5220008Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5220350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5220645Z res = mod(**inputs) 2025-11-03T16:33:43.5220980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5221335Z outputs = self.fnet( 2025-11-03T16:33:43.5221669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5222027Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5222371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5222743Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5223098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5223440Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5223795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5224181Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5224585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5224952Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5225312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5225691Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5225847Z 2025-11-03T16:33:43.5225947Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5226287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5226589Z res = mod(**inputs) 2025-11-03T16:33:43.5226934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5227279Z outputs = self.fnet( 2025-11-03T16:33:43.5227611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5227970Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5228323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5228684Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5229030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5229371Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5229728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5230107Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5230475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5230838Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5231202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5231582Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5231728Z 2025-11-03T16:33:43.5231832Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5232160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5232462Z res = mod(**inputs) 2025-11-03T16:33:43.5232823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5233182Z outputs = self.fnet( 2025-11-03T16:33:43.5233503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5233858Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5234297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5234723Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5235088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5235455Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5235863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5236264Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5236652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5237034Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5237403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5237817Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5237975Z 2025-11-03T16:33:43.5238075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5238424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5238729Z res = mod(**inputs) 2025-11-03T16:33:43.5239063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5239427Z outputs = self.fnet( 2025-11-03T16:33:43.5239765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5240148Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5240500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5240877Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5241230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5241577Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5241944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5242323Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5242710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5243082Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5243450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5243835Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5243987Z 2025-11-03T16:33:43.5244085Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5244437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5244745Z res = mod(**inputs) 2025-11-03T16:33:43.5245078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5245431Z outputs = self.fnet( 2025-11-03T16:33:43.5245802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5246168Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5246526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5246902Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5247247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5247595Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5247966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5248340Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5248720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5249100Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5249495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5249927Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5250334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5250692Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5250848Z 2025-11-03T16:33:43.5250953Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5251280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5251584Z res = mod(**inputs) 2025-11-03T16:33:43.5251910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5252261Z outputs = self.fnet( 2025-11-03T16:33:43.5252591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5252948Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5253310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5253681Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5254024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5254355Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5254712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5255073Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5255449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5255822Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5256198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5256620Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5257010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5257403Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5257758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5258174Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5258392Z 2025-11-03T16:33:43.5258488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5258849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5259160Z res = mod(**inputs) 2025-11-03T16:33:43.5259495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5259855Z outputs = self.fnet( 2025-11-03T16:33:43.5260190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5260558Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5260915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5261290Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5261631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5261974Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5262339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5262709Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5263094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5263459Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5263846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5264314Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5264729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5265102Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5265230Z 2025-11-03T16:33:43.5265309Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5265537Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5265871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5266186Z res = mod(**inputs) 2025-11-03T16:33:43.5266509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5266858Z outputs = self.fnet( 2025-11-03T16:33:43.5267192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5267549Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5267897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5268253Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5268593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5268932Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5269290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5269670Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5270039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5270404Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5270761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5271142Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5271285Z 2025-11-03T16:33:43.5271382Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5271715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5272048Z res = mod(**inputs) 2025-11-03T16:33:43.5272387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5272749Z outputs = self.fnet( 2025-11-03T16:33:43.5273081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5273450Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5273808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5274264Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5274619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5274984Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5275374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5275779Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5276183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5276562Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5276945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5277381Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5277536Z 2025-11-03T16:33:43.5277647Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5277999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5278310Z res = mod(**inputs) 2025-11-03T16:33:43.5278659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5279032Z outputs = self.fnet( 2025-11-03T16:33:43.5279384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5279777Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5280143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5280539Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5280899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5281253Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5281622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5282024Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5282419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5282801Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5283178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5283573Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5283731Z 2025-11-03T16:33:43.5283831Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5284220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5284544Z res = mod(**inputs) 2025-11-03T16:33:43.5284881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5285233Z outputs = self.fnet( 2025-11-03T16:33:43.5285600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5285963Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5286324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5286692Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5287044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5287391Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5287755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5288140Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5288518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5288890Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5289258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5289649Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5289798Z 2025-11-03T16:33:43.5289906Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5290245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5290573Z res = mod(**inputs) 2025-11-03T16:33:43.5290908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5291268Z outputs = self.fnet( 2025-11-03T16:33:43.5291600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5291979Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5292330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5292715Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5293058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5293396Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5293759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5294130Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5294508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5294881Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5295260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5295684Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5296078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5296449Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5296577Z 2025-11-03T16:33:43.5296673Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5297009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5297312Z res = mod(**inputs) 2025-11-03T16:33:43.5297641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5297992Z outputs = self.fnet( 2025-11-03T16:33:43.5298320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5298707Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5299060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5299429Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5299770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5300108Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5300465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5300830Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5301205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5301568Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5301957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5302379Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5302777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5303168Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5303532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5303956Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5304177Z 2025-11-03T16:33:43.5304275Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5304612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5304910Z res = mod(**inputs) 2025-11-03T16:33:43.5305232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5305602Z outputs = self.fnet( 2025-11-03T16:33:43.5305934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5306290Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5306634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5307001Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5307343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5307677Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5308029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5308388Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5308760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5309126Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5309505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5309934Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5310375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5310750Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5310888Z 2025-11-03T16:33:43.5310968Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5311196Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5311576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5311878Z res = mod(**inputs) 2025-11-03T16:33:43.5312226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5312599Z outputs = self.fnet( 2025-11-03T16:33:43.5312949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5313450Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5313815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5314268Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5314636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5315005Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5315372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5315768Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5316156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5316533Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5316938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5317330Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5317486Z 2025-11-03T16:33:43.5317584Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5317920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5318227Z res = mod(**inputs) 2025-11-03T16:33:43.5318560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5318923Z outputs = self.fnet( 2025-11-03T16:33:43.5319283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5319652Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5320011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5320387Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5320737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5321084Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5321462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5321856Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5322244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5322618Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5322988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5323383Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5323534Z 2025-11-03T16:33:43.5323635Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5323972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5324279Z res = mod(**inputs) 2025-11-03T16:33:43.5324620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5324980Z outputs = self.fnet( 2025-11-03T16:33:43.5325357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5325734Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5326085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5326451Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5326662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5326745Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5326974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5327065Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5327304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5327381Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5327616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5327711Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5327715Z 2025-11-03T16:33:43.5327811Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5328018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5328078Z res = mod(**inputs) 2025-11-03T16:33:43.5328311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5328373Z outputs = self.fnet( 2025-11-03T16:33:43.5328604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5328675Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5328909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5329013Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5329221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5329303Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5329536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5329626Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5329868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5329942Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5330184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5330279Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5330283Z 2025-11-03T16:33:43.5330379Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5330564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5330625Z res = mod(**inputs) 2025-11-03T16:33:43.5330864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5330924Z outputs = self.fnet( 2025-11-03T16:33:43.5331159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5331224Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5331483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5331569Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5331774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5331857Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5332084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5332165Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5332415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5332489Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5332756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5332867Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5333100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5333180Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5333184Z 2025-11-03T16:33:43.5333279Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5333472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5333549Z res = mod(**inputs) 2025-11-03T16:33:43.5333785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5333846Z outputs = self.fnet( 2025-11-03T16:33:43.5334073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5334149Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5334380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5334465Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5334682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5334761Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5334987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5335067Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5335314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5335386Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5335649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5335758Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5335988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5336100Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5336300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5336481Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5336485Z 2025-11-03T16:33:43.5336579Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5336770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5336832Z res = mod(**inputs) 2025-11-03T16:33:43.5337092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5337162Z outputs = self.fnet( 2025-11-03T16:33:43.5337390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5337464Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5337690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5337768Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5337977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5338049Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5338283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5338359Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5338602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5338680Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5338937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5339061Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5339309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5339391Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5339394Z 2025-11-03T16:33:43.5339471Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5339564Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5339754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5339816Z res = mod(**inputs) 2025-11-03T16:33:43.5340051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5340128Z outputs = self.fnet( 2025-11-03T16:33:43.5340354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5340428Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5340658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5340743Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5340949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5341028Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5341257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5341348Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5341580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5341657Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5341892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5341989Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5341992Z 2025-11-03T16:33:43.5342088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5342278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5342339Z res = mod(**inputs) 2025-11-03T16:33:43.5342618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5342681Z outputs = self.fnet( 2025-11-03T16:33:43.5342906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5342981Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5343205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5343292Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5343497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5343575Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5343801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5343891Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5344128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5344202Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5344435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5344527Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5344546Z 2025-11-03T16:33:43.5344647Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5344828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5344887Z res = mod(**inputs) 2025-11-03T16:33:43.5345121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5345182Z outputs = self.fnet( 2025-11-03T16:33:43.5345415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5345482Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5345707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5345808Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5346011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5346091Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5346318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5346406Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5346639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5346715Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5346948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5347043Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5347046Z 2025-11-03T16:33:43.5347145Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5347327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5347388Z res = mod(**inputs) 2025-11-03T16:33:43.5347622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5347682Z outputs = self.fnet( 2025-11-03T16:33:43.5347916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5347982Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5348241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5348330Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5348537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5348614Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5348845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5348936Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5349175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5349249Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5349488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5349583Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5349587Z 2025-11-03T16:33:43.5349688Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5349874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5349934Z res = mod(**inputs) 2025-11-03T16:33:43.5350173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5350259Z outputs = self.fnet( 2025-11-03T16:33:43.5350496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5350565Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5350798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5350886Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5351092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5351185Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5351414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5351498Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5351740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5351812Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5352075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5352179Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5352416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5352492Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5352497Z 2025-11-03T16:33:43.5352591Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5352784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5352844Z res = mod(**inputs) 2025-11-03T16:33:43.5353093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5353154Z outputs = self.fnet( 2025-11-03T16:33:43.5353393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5353460Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5353715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5353800Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5354078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5354170Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5354406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5354487Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5354745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5354821Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5355112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5355225Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5355476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5355595Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5355808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5355991Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5356013Z 2025-11-03T16:33:43.5356115Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5356330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5356392Z res = mod(**inputs) 2025-11-03T16:33:43.5356626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5356697Z outputs = self.fnet( 2025-11-03T16:33:43.5356932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5357058Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5357294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5357375Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5357594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5357667Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5357905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5357984Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5358241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5358313Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5358578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5358706Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5358942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5359027Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5359030Z 2025-11-03T16:33:43.5359107Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5359205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5359401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5359461Z res = mod(**inputs) 2025-11-03T16:33:43.5359730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5359796Z outputs = self.fnet( 2025-11-03T16:33:43.5360040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5360109Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5360343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5360433Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5360644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5360725Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5360962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5361059Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5361300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5361379Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5361619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5361716Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5361734Z 2025-11-03T16:33:43.5361839Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5362028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5362090Z res = mod(**inputs) 2025-11-03T16:33:43.5362335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5362397Z outputs = self.fnet( 2025-11-03T16:33:43.5362641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5362710Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5362969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5363057Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5363270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5363354Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5363587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5363680Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5363923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5364001Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5364245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5364343Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5364346Z 2025-11-03T16:33:43.5364449Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5364639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5364700Z res = mod(**inputs) 2025-11-03T16:33:43.5364942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5365004Z outputs = self.fnet( 2025-11-03T16:33:43.5365247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5365344Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5365582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5365671Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5365884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5365964Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5366199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5366298Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5366534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5366610Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5366852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5366949Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5366954Z 2025-11-03T16:33:43.5367057Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5367248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5367308Z res = mod(**inputs) 2025-11-03T16:33:43.5367568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5367631Z outputs = self.fnet( 2025-11-03T16:33:43.5367873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5367942Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5368181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5368266Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5368481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5368580Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5368816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5368916Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5369151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5369226Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5369473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5369569Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5369575Z 2025-11-03T16:33:43.5369678Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5369866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5369929Z res = mod(**inputs) 2025-11-03T16:33:43.5370170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5370234Z outputs = self.fnet( 2025-11-03T16:33:43.5370475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5370543Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5370786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5370865Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5371108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5371193Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5371424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5371508Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5371752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5371826Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5372095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5372202Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5372441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5372522Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5372525Z 2025-11-03T16:33:43.5372626Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5372819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5372879Z res = mod(**inputs) 2025-11-03T16:33:43.5373117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5373196Z outputs = self.fnet( 2025-11-03T16:33:43.5373442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5373511Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5373750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5373837Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5374052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5374133Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5374379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5374458Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5374710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5374783Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5375054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5375162Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5375404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5375507Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5375712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5375891Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5375896Z 2025-11-03T16:33:43.5375994Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5376191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5376253Z res = mod(**inputs) 2025-11-03T16:33:43.5376494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5376558Z outputs = self.fnet( 2025-11-03T16:33:43.5376819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5376897Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5377133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5377223Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5377436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5377511Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5377755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5377834Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5378088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5378163Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5378433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5378563Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5378801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5378905Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5378909Z 2025-11-03T16:33:43.5378986Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5379091Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5379279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5379340Z res = mod(**inputs) 2025-11-03T16:33:43.5379580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5379646Z outputs = self.fnet( 2025-11-03T16:33:43.5379885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5379971Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5380207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5380298Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5380512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5380592Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5380824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5380925Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5381162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5381237Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5381479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5381575Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5381579Z 2025-11-03T16:33:43.5381684Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5381872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5381932Z res = mod(**inputs) 2025-11-03T16:33:43.5382175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5382239Z outputs = self.fnet( 2025-11-03T16:33:43.5382511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5382581Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5382820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5382910Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5383128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5383213Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5383455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5383556Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5383800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5383879Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5384142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5384241Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5384244Z 2025-11-03T16:33:43.5384348Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5384538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5384630Z res = mod(**inputs) 2025-11-03T16:33:43.5384876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5384941Z outputs = self.fnet( 2025-11-03T16:33:43.5385190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5385261Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5385511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5385593Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5385826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5385909Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5386149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5386252Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5386493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5386570Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5386817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5386916Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5386920Z 2025-11-03T16:33:43.5387025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5387217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5387286Z res = mod(**inputs) 2025-11-03T16:33:43.5387527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5387593Z outputs = self.fnet( 2025-11-03T16:33:43.5387841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5387912Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5388159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5388241Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5388485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5388573Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5388814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5388914Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5389156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5389236Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5389482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5389580Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5389583Z 2025-11-03T16:33:43.5389692Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5389883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5389956Z res = mod(**inputs) 2025-11-03T16:33:43.5390197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5390262Z outputs = self.fnet( 2025-11-03T16:33:43.5390509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5390597Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5390851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5390934Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5391152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5391238Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5391479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5391583Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5391838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5391920Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5392199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5392310Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5392561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5392653Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5392656Z 2025-11-03T16:33:43.5392759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5392947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5393009Z res = mod(**inputs) 2025-11-03T16:33:43.5393246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5393308Z outputs = self.fnet( 2025-11-03T16:33:43.5393550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5393617Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5393852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5393940Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5394254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5394345Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5394580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5394671Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5394919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5394994Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5395268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5395379Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5395628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5395732Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5395933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5396112Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5396115Z 2025-11-03T16:33:43.5396213Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5396427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5396488Z res = mod(**inputs) 2025-11-03T16:33:43.5396723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5396786Z outputs = self.fnet( 2025-11-03T16:33:43.5397015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5397095Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5397320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5397929Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5398134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5398207Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5398440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5398516Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5398762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5398832Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5399096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5399214Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5399446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5399530Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5399533Z 2025-11-03T16:33:43.5399610Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5399710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5399892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5399951Z res = mod(**inputs) 2025-11-03T16:33:43.5400185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5400244Z outputs = self.fnet( 2025-11-03T16:33:43.5400507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5400578Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5400812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5400890Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5401092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5401175Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5401400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5401497Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5401727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5401804Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5402041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5402137Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5402140Z 2025-11-03T16:33:43.5402240Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5402435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5402494Z res = mod(**inputs) 2025-11-03T16:33:43.5402734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5402793Z outputs = self.fnet( 2025-11-03T16:33:43.5403028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5403095Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5403334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5403430Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5403635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5403716Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5403947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5404045Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5404272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5404346Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5404586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5404678Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5404682Z 2025-11-03T16:33:43.5404785Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5404967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5405035Z res = mod(**inputs) 2025-11-03T16:33:43.5405266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5405328Z outputs = self.fnet( 2025-11-03T16:33:43.5405567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5405633Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5405868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5405982Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5406188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5406271Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5406497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5406595Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5406821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5406897Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5407138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5407232Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5407235Z 2025-11-03T16:33:43.5407340Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5407521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5407589Z res = mod(**inputs) 2025-11-03T16:33:43.5407815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5407894Z outputs = self.fnet( 2025-11-03T16:33:43.5408128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5408195Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5408425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5408499Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5408707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5408784Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5409009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5409114Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5409343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5409422Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5409648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5409737Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5409740Z 2025-11-03T16:33:43.5409842Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5410025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5410090Z res = mod(**inputs) 2025-11-03T16:33:43.5410321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5410382Z outputs = self.fnet( 2025-11-03T16:33:43.5410618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5410686Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5410919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5410992Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5411204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5411277Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5411535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5411622Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5411866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5411944Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5412205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5412315Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5412552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5412627Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5412630Z 2025-11-03T16:33:43.5412730Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5412918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5412983Z res = mod(**inputs) 2025-11-03T16:33:43.5413328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5413402Z outputs = self.fnet( 2025-11-03T16:33:43.5413641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5413754Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5413990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5414069Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5414274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5414358Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5414591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5414699Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5414938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5415010Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5415273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5415378Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5415615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5415717Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5415925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5416093Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5416099Z 2025-11-03T16:33:43.5416194Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5416384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5416446Z res = mod(**inputs) 2025-11-03T16:33:43.5416682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5416744Z outputs = self.fnet( 2025-11-03T16:33:43.5416970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5417044Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5417315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5417402Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5417610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5417689Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5417919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5417996Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5418243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5418314Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5418579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5418699Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5418926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5419010Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5419013Z 2025-11-03T16:33:43.5419090Z cudagraph partition due to non gpu ops 2025-11-03T16:33:43.5419190Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5419385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5419444Z res = mod(**inputs) 2025-11-03T16:33:43.5419676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5419734Z outputs = self.fnet( 2025-11-03T16:33:43.5419967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5420035Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5420278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5420372Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5420576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5420682Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5420908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5421004Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5421233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5421307Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5421544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5421639Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5421643Z 2025-11-03T16:33:43.5421744Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5421926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5421995Z res = mod(**inputs) 2025-11-03T16:33:43.5422224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5422284Z outputs = self.fnet( 2025-11-03T16:33:43.5422518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5422585Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5422847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5422926Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5423131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5423211Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5423438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5423535Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5423765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5423847Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5424075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5424167Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5424170Z 2025-11-03T16:33:43.5424269Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5424452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5424518Z res = mod(**inputs) 2025-11-03T16:33:43.5424748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5424826Z outputs = self.fnet( 2025-11-03T16:33:43.5425071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5425139Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5425374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5425452Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5425664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5425744Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5425986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5426082Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5426314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5426397Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5426625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5426718Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5426721Z 2025-11-03T16:33:43.5426824Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5427007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5427075Z res = mod(**inputs) 2025-11-03T16:33:43.5427300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5427359Z outputs = self.fnet( 2025-11-03T16:33:43.5427588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5427653Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5427885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5427962Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5428168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5428276Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5428504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-11-03T16:33:43.5428596Z self_fourier_outputs = self.fourier(hidden_states) 2025-11-03T16:33:43.5428821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-11-03T16:33:43.5428901Z self_outputs = self.self(hidden_states) 2025-11-03T16:33:43.5429130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-11-03T16:33:43.5429223Z outputs = self.fourier_transform(hidden_states).real 2025-11-03T16:33:43.5429234Z 2025-11-03T16:33:43.5429327Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5429506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5429572Z res = mod(**inputs) 2025-11-03T16:33:43.5429801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5429870Z outputs = self.fnet( 2025-11-03T16:33:43.5430098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5430164Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5430416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5430493Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5430703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5430775Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5431000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5431088Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5431330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5431420Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5431677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5431781Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5432023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-11-03T16:33:43.5432099Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5432103Z 2025-11-03T16:33:43.5432201Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5432390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5432458Z res = mod(**inputs) 2025-11-03T16:33:43.5432688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5432752Z outputs = self.fnet( 2025-11-03T16:33:43.5432989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5433058Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5433294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5433373Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5433580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5433662Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5433933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5434084Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5434355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5434439Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5434714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-11-03T16:33:43.5434828Z intermediate_output = self.intermediate(fourier_output) 2025-11-03T16:33:43.5435139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-11-03T16:33:43.5435245Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:33:43.5435454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 48, in forward 2025-11-03T16:33:43.5435628Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-11-03T16:33:43.5435632Z 2025-11-03T16:33:43.5435738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5435925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5435987Z res = mod(**inputs) 2025-11-03T16:33:43.5436245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-11-03T16:33:43.5436303Z outputs = self.fnet( 2025-11-03T16:33:43.5436548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-11-03T16:33:43.5436619Z encoder_outputs = self.encoder( 2025-11-03T16:33:43.5436850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-11-03T16:33:43.5436939Z layer_outputs = layer_module(hidden_states) 2025-11-03T16:33:43.5437144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:33:43.5437239Z return super().__call__(*args, **kwargs) 2025-11-03T16:33:43.5437469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-11-03T16:33:43.5437547Z layer_output = apply_chunking_to_forward( 2025-11-03T16:33:43.5437791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:33:43.5437862Z return forward_fn(*input_tensors) 2025-11-03T16:33:43.5438125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-11-03T16:33:43.5438243Z layer_output = self.output(intermediate_output, fourier_output) 2025-11-03T16:33:43.5438480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-11-03T16:33:43.5438557Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5438562Z 2025-11-03T16:33:43.5438655Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5438847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5438908Z res = mod(**inputs) 2025-11-03T16:33:43.5439145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-11-03T16:33:43.5439233Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:33:43.5439462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-11-03T16:33:43.5439573Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:33:43.5439829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-11-03T16:33:43.5439924Z hidden_states = self.transform(hidden_states) 2025-11-03T16:33:43.5440157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 321, in forward 2025-11-03T16:33:43.5440241Z hidden_states = self.dense(hidden_states) 2025-11-03T16:33:43.5440245Z 2025-11-03T16:33:43.5440341Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5440524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5440593Z res = mod(**inputs) 2025-11-03T16:33:43.5440824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-11-03T16:33:43.5440916Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:33:43.5441148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-11-03T16:33:43.5441251Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:33:43.5441488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 341, in forward 2025-11-03T16:33:43.5441568Z hidden_states = self.decoder(hidden_states) 2025-11-03T16:33:43.5441572Z 2025-11-03T16:33:43.5441695Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:33:43.5441876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:33:43.5441940Z res = mod(**inputs) 2025-11-03T16:33:43.5442170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 686, in forward 2025-11-03T16:33:43.5442345Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:33:43.5442348Z 2025-11-03T16:33:52.6567095Z Compilation time (from dynamo_timed): 13.642991121 2025-11-03T16:33:52.6614336Z pass 2025-11-03T16:33:52.6618531Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:52.6623864Z TIMING: _recursive_pre_grad_passes:0.00614 _recursive_joint_graph_passes:0.20568 _recursive_post_grad_passes:0.05857 async_compile.wait:0.71688 code_gen:8.80304 inductor_compile:9.94764 backend_compile:11.70149 gc:0.00033 entire_frame_compile:13.64299 total_wall_time:13.64299 2025-11-03T16:33:52.6625454Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:4171 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:1233 2025-11-03T16:33:52.6625972Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-11-03T16:33:54.8799014Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:33:54.8800088Z import pynvml # type: ignore[import] 2025-11-03T16:33:57.9620609Z 2025-11-03T16:33:59.0651594Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:33:59.0656153Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:33:59.0662862Z cpu eval LayoutLMForMaskedLM 2025-11-03T16:33:59.6693473Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:33:59.9145656Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:00.1620359Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:07.8244659Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8249660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8254693Z res = mod(**inputs) 2025-11-03T16:34:07.8259291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8261352Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8261979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8267146Z outputs = self.layoutlm( 2025-11-03T16:34:07.8272071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8274259Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8274914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8280630Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8283033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8283592Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8288252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8288854Z layer_outputs = layer_module( 2025-11-03T16:34:07.8296075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8298083Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8298557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8303840Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8309492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8314207Z self_outputs = self.self( 2025-11-03T16:34:07.8319182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8321370Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8326284Z 2025-11-03T16:34:07.8331893Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8337374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8339509Z res = mod(**inputs) 2025-11-03T16:34:07.8340008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8340389Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8340808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8341208Z outputs = self.layoutlm( 2025-11-03T16:34:07.8341566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8341919Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8342324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8342727Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8343068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8343414Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8343814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8344264Z layer_outputs = layer_module( 2025-11-03T16:34:07.8344601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8345169Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8345591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8346008Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8346427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8346814Z self_outputs = self.self( 2025-11-03T16:34:07.8347195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8347641Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8347834Z 2025-11-03T16:34:07.8347964Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8348340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8348665Z res = mod(**inputs) 2025-11-03T16:34:07.8349003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8349356Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8349753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8350142Z outputs = self.layoutlm( 2025-11-03T16:34:07.8350524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8350876Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8351276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8351682Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8352027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8352457Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8352856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8353282Z layer_outputs = layer_module( 2025-11-03T16:34:07.8353640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8354124Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8354571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8355014Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8355507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8355894Z self_outputs = self.self( 2025-11-03T16:34:07.8356299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8356773Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8356985Z 2025-11-03T16:34:07.8357069Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8357285Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8357525Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8357895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8358215Z res = mod(**inputs) 2025-11-03T16:34:07.8358536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8358914Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8359357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8359753Z outputs = self.layoutlm( 2025-11-03T16:34:07.8360090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8360447Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8360846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8361242Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8361590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8361941Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8362339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8362738Z layer_outputs = layer_module( 2025-11-03T16:34:07.8363082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8363447Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8363856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8364263Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8364668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8365136Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8365596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8365992Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8366131Z 2025-11-03T16:34:07.8366240Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8366590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8366917Z res = mod(**inputs) 2025-11-03T16:34:07.8367249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8367599Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8367991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8368377Z outputs = self.layoutlm( 2025-11-03T16:34:07.8369066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8369414Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8369808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8370204Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8370540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8370887Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8371278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8371672Z layer_outputs = layer_module( 2025-11-03T16:34:07.8372010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8372368Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8372770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8373178Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8373612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8374002Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8374430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8374910Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8375359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8375771Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8375911Z 2025-11-03T16:34:07.8376015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8376376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8376698Z res = mod(**inputs) 2025-11-03T16:34:07.8377026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8377378Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8377771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8378171Z outputs = self.layoutlm( 2025-11-03T16:34:07.8378506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8378875Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8379267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8379667Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8380010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8380359Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8380759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8381147Z layer_outputs = layer_module( 2025-11-03T16:34:07.8381504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8381864Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8382263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8382673Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8383065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8383454Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8383881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8384376Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8384811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8385243Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8385626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8385969Z return self.act(input) 2025-11-03T16:34:07.8386081Z 2025-11-03T16:34:07.8386189Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8386537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8386854Z res = mod(**inputs) 2025-11-03T16:34:07.8387176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8387575Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8387976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8388367Z outputs = self.layoutlm( 2025-11-03T16:34:07.8388703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8389053Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8389451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8389843Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8390191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8390538Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8390933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8391326Z layer_outputs = layer_module( 2025-11-03T16:34:07.8391662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8392027Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8392436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8392871Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8393272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8393659Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8394187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8394714Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8395201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8395620Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8395755Z 2025-11-03T16:34:07.8395859Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8396208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8396522Z res = mod(**inputs) 2025-11-03T16:34:07.8396836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8397173Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8397560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8397943Z outputs = self.layoutlm( 2025-11-03T16:34:07.8398274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8398615Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8398999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8399384Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8399718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8400056Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8400431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8400810Z layer_outputs = layer_module( 2025-11-03T16:34:07.8401145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8401524Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8401912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8402301Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8402693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8403077Z self_outputs = self.self( 2025-11-03T16:34:07.8403449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8403903Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8404097Z 2025-11-03T16:34:07.8404196Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8404543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8404856Z res = mod(**inputs) 2025-11-03T16:34:07.8405170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8405506Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8405889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8406290Z outputs = self.layoutlm( 2025-11-03T16:34:07.8406616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8406965Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8407336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8407713Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8408043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8408375Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8408754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8409143Z layer_outputs = layer_module( 2025-11-03T16:34:07.8409474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8409826Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8410211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8410593Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8410977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8411352Z self_outputs = self.self( 2025-11-03T16:34:07.8411724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8412164Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8412344Z 2025-11-03T16:34:07.8412443Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8412784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8413089Z res = mod(**inputs) 2025-11-03T16:34:07.8413719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8414062Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8414437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8414815Z outputs = self.layoutlm( 2025-11-03T16:34:07.8415197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8415528Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8415897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8416270Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8416594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8416932Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8417304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8417668Z layer_outputs = layer_module( 2025-11-03T16:34:07.8417995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8418341Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8418721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8419105Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8419478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8419877Z self_outputs = self.self( 2025-11-03T16:34:07.8420244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8420688Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8420878Z 2025-11-03T16:34:07.8420960Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8421157Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8421381Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8421725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8422030Z res = mod(**inputs) 2025-11-03T16:34:07.8422357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8422690Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8423065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8423443Z outputs = self.layoutlm( 2025-11-03T16:34:07.8423751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8424082Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8424456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8424838Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8425164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8425489Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8425870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8426254Z layer_outputs = layer_module( 2025-11-03T16:34:07.8426582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8426925Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8427297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8427681Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8428092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8428524Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8428952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8429331Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8429467Z 2025-11-03T16:34:07.8429567Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8429902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8430205Z res = mod(**inputs) 2025-11-03T16:34:07.8430501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8430831Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8431207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8431580Z outputs = self.layoutlm( 2025-11-03T16:34:07.8431897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8432223Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8432603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8432993Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8433315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8433634Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8434084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8434480Z layer_outputs = layer_module( 2025-11-03T16:34:07.8434825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8435189Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8435622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8436033Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8436438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8436835Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8437259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8437737Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8438186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8438592Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8438728Z 2025-11-03T16:34:07.8438838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8439194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8439508Z res = mod(**inputs) 2025-11-03T16:34:07.8439829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8440179Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8440572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8440960Z outputs = self.layoutlm( 2025-11-03T16:34:07.8441295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8441690Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8442085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8442626Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8442970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8443325Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8443734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8444137Z layer_outputs = layer_module( 2025-11-03T16:34:07.8444475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8444831Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8445219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8445613Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8445998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8446367Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8446772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8447243Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8447663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8448078Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8448432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8448762Z return self.act(input) 2025-11-03T16:34:07.8448877Z 2025-11-03T16:34:07.8448974Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8449329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8449636Z res = mod(**inputs) 2025-11-03T16:34:07.8449937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8450278Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8450655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8451029Z outputs = self.layoutlm( 2025-11-03T16:34:07.8451334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8451666Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8452042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8452417Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8452742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8453063Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8453438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8453813Z layer_outputs = layer_module( 2025-11-03T16:34:07.8454142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8454482Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8454852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8455272Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8455654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8456022Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8456419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8456881Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8457311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8457700Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8457829Z 2025-11-03T16:34:07.8457933Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8458268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8458576Z res = mod(**inputs) 2025-11-03T16:34:07.8458880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8459216Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8459591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8459982Z outputs = self.layoutlm( 2025-11-03T16:34:07.8460302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8460634Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8461010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8461381Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8461717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8462046Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8462423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8462812Z layer_outputs = layer_module( 2025-11-03T16:34:07.8463131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8463477Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8463888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8464291Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8464691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8465075Z self_outputs = self.self( 2025-11-03T16:34:07.8465456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8465920Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8466109Z 2025-11-03T16:34:07.8466214Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8466553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8466852Z res = mod(**inputs) 2025-11-03T16:34:07.8467160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8467494Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8467869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8468235Z outputs = self.layoutlm( 2025-11-03T16:34:07.8468583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8468914Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8469295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8469670Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8470025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8470365Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8470749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8471176Z layer_outputs = layer_module( 2025-11-03T16:34:07.8471504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8471859Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8472251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8472652Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8473052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8473452Z self_outputs = self.self( 2025-11-03T16:34:07.8473838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8474364Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8474559Z 2025-11-03T16:34:07.8474670Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8475031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8475350Z res = mod(**inputs) 2025-11-03T16:34:07.8475677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8476050Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8476451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8476842Z outputs = self.layoutlm( 2025-11-03T16:34:07.8477177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8477532Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8477931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8478328Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8478671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8479020Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8479413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8479814Z layer_outputs = layer_module( 2025-11-03T16:34:07.8480161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8480513Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8480913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8481319Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8481718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8482103Z self_outputs = self.self( 2025-11-03T16:34:07.8482545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8483019Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8483220Z 2025-11-03T16:34:07.8483310Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8483521Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8483747Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8484106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8484426Z res = mod(**inputs) 2025-11-03T16:34:07.8484758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8485097Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8485484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8485868Z outputs = self.layoutlm( 2025-11-03T16:34:07.8486192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8486536Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8486910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8487318Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8487653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8487996Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8488383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8488769Z layer_outputs = layer_module( 2025-11-03T16:34:07.8489116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8489458Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8489869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8490247Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8490644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8491076Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8491507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8491899Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8492033Z 2025-11-03T16:34:07.8492133Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8492481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8492791Z res = mod(**inputs) 2025-11-03T16:34:07.8493109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8493446Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8493818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8494199Z outputs = self.layoutlm( 2025-11-03T16:34:07.8494523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8494860Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8495230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8495641Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8495968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8496297Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8496669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8497033Z layer_outputs = layer_module( 2025-11-03T16:34:07.8497359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8497700Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8498078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8498465Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8498837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8499207Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8499611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8500059Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8500475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8500875Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8501012Z 2025-11-03T16:34:07.8501111Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8501447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8501751Z res = mod(**inputs) 2025-11-03T16:34:07.8502055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8502390Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8502766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8503161Z outputs = self.layoutlm( 2025-11-03T16:34:07.8503479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8503803Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8504177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8504562Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8504889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8505218Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8505589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8505962Z layer_outputs = layer_module( 2025-11-03T16:34:07.8506288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8506632Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8507007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8507398Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8507777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8508149Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8508584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8509026Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8509450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8509867Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8510229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8510554Z return self.act(input) 2025-11-03T16:34:07.8510658Z 2025-11-03T16:34:07.8510757Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8511103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8511413Z res = mod(**inputs) 2025-11-03T16:34:07.8511726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8512062Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8512448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8512834Z outputs = self.layoutlm( 2025-11-03T16:34:07.8513156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8513662Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8514139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8514552Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8514912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8515268Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8515712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8516099Z layer_outputs = layer_module( 2025-11-03T16:34:07.8516440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8516868Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8517259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8517652Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8518038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8518415Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8518832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8519305Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8519747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8520150Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8520293Z 2025-11-03T16:34:07.8520393Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8520740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8521055Z res = mod(**inputs) 2025-11-03T16:34:07.8521361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8521704Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8522091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8522473Z outputs = self.layoutlm( 2025-11-03T16:34:07.8522836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8523181Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8523569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8523960Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8524302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8524636Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8525023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8525413Z layer_outputs = layer_module( 2025-11-03T16:34:07.8525750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8526115Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8526496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8526890Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8527280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8527682Z self_outputs = self.self( 2025-11-03T16:34:07.8528044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8528493Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8528690Z 2025-11-03T16:34:07.8528789Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8529132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8529440Z res = mod(**inputs) 2025-11-03T16:34:07.8529743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8530090Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8530472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8530849Z outputs = self.layoutlm( 2025-11-03T16:34:07.8531163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8531485Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8531860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8532235Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8532565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8532884Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8533259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8533642Z layer_outputs = layer_module( 2025-11-03T16:34:07.8533970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8534308Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8534676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8535058Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8535439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8535843Z self_outputs = self.self( 2025-11-03T16:34:07.8536207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8536633Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8536819Z 2025-11-03T16:34:07.8536917Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8537251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8537557Z res = mod(**inputs) 2025-11-03T16:34:07.8537851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8538182Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8538557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8538935Z outputs = self.layoutlm( 2025-11-03T16:34:07.8539250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8539573Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8539948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8540320Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8540689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8541018Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8541386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8541760Z layer_outputs = layer_module( 2025-11-03T16:34:07.8542097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8542449Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8542832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8543243Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8543631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8544024Z self_outputs = self.self( 2025-11-03T16:34:07.8544387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8544833Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8545031Z 2025-11-03T16:34:07.8545112Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8545317Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8545544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8545893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8546198Z res = mod(**inputs) 2025-11-03T16:34:07.8546511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8546855Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8547238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8547618Z outputs = self.layoutlm( 2025-11-03T16:34:07.8547942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8548279Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8548662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8549078Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8549413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8549762Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8550152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8550549Z layer_outputs = layer_module( 2025-11-03T16:34:07.8550885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8551242Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8551640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8552045Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8552449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8552892Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8553346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8553750Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8553896Z 2025-11-03T16:34:07.8554075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8554431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8554737Z res = mod(**inputs) 2025-11-03T16:34:07.8555058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8555421Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8555804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8556176Z outputs = self.layoutlm( 2025-11-03T16:34:07.8556502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8556863Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8557255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8557642Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8557968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8558310Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8558693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8559081Z layer_outputs = layer_module( 2025-11-03T16:34:07.8559432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8559776Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8560169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8560569Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8560959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8561332Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8561754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8562214Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8562681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8563082Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8563218Z 2025-11-03T16:34:07.8563315Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8563662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8563976Z res = mod(**inputs) 2025-11-03T16:34:07.8564294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8564634Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8565028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8565425Z outputs = self.layoutlm( 2025-11-03T16:34:07.8565760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8566113Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8566502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8566902Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8567245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8567614Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8568007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8568393Z layer_outputs = layer_module( 2025-11-03T16:34:07.8568741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8569101Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8569506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8569914Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8570320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8570709Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8571146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8571624Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8572069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8572497Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8572882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8573222Z return self.act(input) 2025-11-03T16:34:07.8573332Z 2025-11-03T16:34:07.8573441Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8573795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8574123Z res = mod(**inputs) 2025-11-03T16:34:07.8574446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8574801Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8575203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8575591Z outputs = self.layoutlm( 2025-11-03T16:34:07.8575924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8576276Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8576706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8577109Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8577446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8577790Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8578183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8578584Z layer_outputs = layer_module( 2025-11-03T16:34:07.8578924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8579289Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8579701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8580096Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8580480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8580851Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8581266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8581763Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8582206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8582620Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8582756Z 2025-11-03T16:34:07.8582860Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8583225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8583549Z res = mod(**inputs) 2025-11-03T16:34:07.8583871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8584243Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8584632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8585018Z outputs = self.layoutlm( 2025-11-03T16:34:07.8585345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8585685Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8586065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8586451Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8586789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8587125Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8587509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8587891Z layer_outputs = layer_module( 2025-11-03T16:34:07.8588224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8588578Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8588970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8589359Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8589756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8590191Z self_outputs = self.self( 2025-11-03T16:34:07.8590572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8591032Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8591225Z 2025-11-03T16:34:07.8591327Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8591677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8591989Z res = mod(**inputs) 2025-11-03T16:34:07.8592302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8592645Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8593020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8593413Z outputs = self.layoutlm( 2025-11-03T16:34:07.8593740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8594200Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8594590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8594989Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8595370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8595712Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8596097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8596475Z layer_outputs = layer_module( 2025-11-03T16:34:07.8596821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8597175Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8597566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8597977Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8598369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8598745Z self_outputs = self.self( 2025-11-03T16:34:07.8599107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8599543Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8599723Z 2025-11-03T16:34:07.8599828Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8600164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8600469Z res = mod(**inputs) 2025-11-03T16:34:07.8600773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8601105Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8601470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8601844Z outputs = self.layoutlm( 2025-11-03T16:34:07.8602158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8602487Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8602859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8603230Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8603584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8603915Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8604291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8604659Z layer_outputs = layer_module( 2025-11-03T16:34:07.8604987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8605331Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8605712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8606103Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8606478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8606851Z self_outputs = self.self( 2025-11-03T16:34:07.8607216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8607669Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8607859Z 2025-11-03T16:34:07.8607944Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8608141Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8608380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8608722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8609029Z res = mod(**inputs) 2025-11-03T16:34:07.8609326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8609662Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8610036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8610407Z outputs = self.layoutlm( 2025-11-03T16:34:07.8610720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8611072Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8611447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8611820Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8612143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8612463Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8612835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8613204Z layer_outputs = layer_module( 2025-11-03T16:34:07.8613657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8614005Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8614391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8614790Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8615186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8615633Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8616072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8616467Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8616610Z 2025-11-03T16:34:07.8616777Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8617128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8617443Z res = mod(**inputs) 2025-11-03T16:34:07.8617749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8618094Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8618480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8618866Z outputs = self.layoutlm( 2025-11-03T16:34:07.8619188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8619525Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8619915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8620313Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8620651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8620994Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8621374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8621787Z layer_outputs = layer_module( 2025-11-03T16:34:07.8622122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8622474Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8622861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8623264Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8623654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8624038Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8624485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8624941Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8625377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8625772Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8625903Z 2025-11-03T16:34:07.8626013Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8626358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8626677Z res = mod(**inputs) 2025-11-03T16:34:07.8627000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8627344Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8627733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8628149Z outputs = self.layoutlm( 2025-11-03T16:34:07.8628465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8628807Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8629201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8629589Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8629940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8630295Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8630728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8631122Z layer_outputs = layer_module( 2025-11-03T16:34:07.8631459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8631806Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8632200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8632602Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8632990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8633368Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8633781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8634314Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8634763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8635193Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8635571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8635931Z return self.act(input) 2025-11-03T16:34:07.8636046Z 2025-11-03T16:34:07.8636147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8636495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8636819Z res = mod(**inputs) 2025-11-03T16:34:07.8637129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8637472Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8637857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8638264Z outputs = self.layoutlm( 2025-11-03T16:34:07.8638583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8638930Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8639321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8639715Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8640053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8640397Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8640792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8641190Z layer_outputs = layer_module( 2025-11-03T16:34:07.8641532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8641888Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8642278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8642687Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8643082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8643473Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8643893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8644411Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8644857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8645269Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8645400Z 2025-11-03T16:34:07.8645508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8645843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8646149Z res = mod(**inputs) 2025-11-03T16:34:07.8646458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8646798Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8647177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8647552Z outputs = self.layoutlm( 2025-11-03T16:34:07.8647876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8648212Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8648590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8648973Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8649307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8649643Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8650031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8650418Z layer_outputs = layer_module( 2025-11-03T16:34:07.8650741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8651082Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8651462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8651863Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8652246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8652614Z self_outputs = self.self( 2025-11-03T16:34:07.8652980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8653419Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8653607Z 2025-11-03T16:34:07.8653715Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8654058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8654356Z res = mod(**inputs) 2025-11-03T16:34:07.8654660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8655001Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8655389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8655769Z outputs = self.layoutlm( 2025-11-03T16:34:07.8656094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8656444Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8656820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8657196Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8657545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8657877Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8658267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8658651Z layer_outputs = layer_module( 2025-11-03T16:34:07.8658984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8659326Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8659715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8660111Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8660505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8660883Z self_outputs = self.self( 2025-11-03T16:34:07.8661259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8661703Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8661886Z 2025-11-03T16:34:07.8661995Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8662359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8662663Z res = mod(**inputs) 2025-11-03T16:34:07.8662979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8663325Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8663712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8664101Z outputs = self.layoutlm( 2025-11-03T16:34:07.8664420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8664776Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8665160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8665543Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8665867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8666202Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8666583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8666967Z layer_outputs = layer_module( 2025-11-03T16:34:07.8667302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8667642Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8668028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8668425Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8668819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8669204Z self_outputs = self.self( 2025-11-03T16:34:07.8669573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8670023Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8670223Z 2025-11-03T16:34:07.8670301Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8670510Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8670759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8671109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8671427Z res = mod(**inputs) 2025-11-03T16:34:07.8671742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8672091Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8672475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8672862Z outputs = self.layoutlm( 2025-11-03T16:34:07.8673188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8673533Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8673925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8674410Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8674764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8675120Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8675512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8675934Z layer_outputs = layer_module( 2025-11-03T16:34:07.8676282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8676649Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8677067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8677466Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8677857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8678297Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8678758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8679160Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8679297Z 2025-11-03T16:34:07.8679409Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8679757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8680083Z res = mod(**inputs) 2025-11-03T16:34:07.8680406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8680760Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8681152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8681547Z outputs = self.layoutlm( 2025-11-03T16:34:07.8681881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8682229Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8682630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8683041Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8683392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8683802Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8684200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8684598Z layer_outputs = layer_module( 2025-11-03T16:34:07.8684981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8685339Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8685740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8686144Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8686536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8686919Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8687345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8687817Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8688260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8688659Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8688804Z 2025-11-03T16:34:07.8688906Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8689256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8689578Z res = mod(**inputs) 2025-11-03T16:34:07.8689914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8690263Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8690662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8691059Z outputs = self.layoutlm( 2025-11-03T16:34:07.8691396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8691742Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8692139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8692635Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8692971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8693314Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8693682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8694070Z layer_outputs = layer_module( 2025-11-03T16:34:07.8694398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8694749Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8695139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8695532Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8695920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8696294Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8696703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8697148Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8697569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8697986Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8698385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8698709Z return self.act(input) 2025-11-03T16:34:07.8698812Z 2025-11-03T16:34:07.8698910Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8699249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8699552Z res = mod(**inputs) 2025-11-03T16:34:07.8699856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8700189Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8700558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8700935Z outputs = self.layoutlm( 2025-11-03T16:34:07.8701252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8701585Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8701954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8702333Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8702655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8702984Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8703378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8703744Z layer_outputs = layer_module( 2025-11-03T16:34:07.8704076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8704415Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8704800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8705186Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8705564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8705949Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8706351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8706809Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8707232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8707318Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8707322Z 2025-11-03T16:34:07.8707422Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8707619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8707683Z res = mod(**inputs) 2025-11-03T16:34:07.8707896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8707968Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8708222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8708299Z outputs = self.layoutlm( 2025-11-03T16:34:07.8708503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8708579Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8708832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8708902Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8710068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8710150Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8710412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8710481Z layer_outputs = layer_module( 2025-11-03T16:34:07.8710691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8710779Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8711035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8711125Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8711383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8711462Z self_outputs = self.self( 2025-11-03T16:34:07.8711723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8711868Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8711872Z 2025-11-03T16:34:07.8711980Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8712187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8712256Z res = mod(**inputs) 2025-11-03T16:34:07.8712466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8712536Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8712800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8712871Z outputs = self.layoutlm( 2025-11-03T16:34:07.8713084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8713171Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8713556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8713635Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8713847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8713928Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8714240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8714331Z layer_outputs = layer_module( 2025-11-03T16:34:07.8714564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8714643Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8714915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8714999Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8715276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8715348Z self_outputs = self.self( 2025-11-03T16:34:07.8715613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8715745Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8715750Z 2025-11-03T16:34:07.8715849Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8716105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8716174Z res = mod(**inputs) 2025-11-03T16:34:07.8716389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8716464Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8716719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8716797Z outputs = self.layoutlm( 2025-11-03T16:34:07.8717000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8717079Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8717332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8717404Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8717621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8717691Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8717957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8718025Z layer_outputs = layer_module( 2025-11-03T16:34:07.8718244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8718388Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8718647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8718732Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8718987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8719063Z self_outputs = self.self( 2025-11-03T16:34:07.8719324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8719485Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8719496Z 2025-11-03T16:34:07.8719577Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8719653Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8719761Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8719951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8720012Z res = mod(**inputs) 2025-11-03T16:34:07.8720226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8720297Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8720560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8720628Z outputs = self.layoutlm( 2025-11-03T16:34:07.8720843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8720913Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8721165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8721245Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8721450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8721529Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8721782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8721877Z layer_outputs = layer_module( 2025-11-03T16:34:07.8722099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8722176Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8722435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8722514Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8722776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8722899Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8723150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8723239Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8723242Z 2025-11-03T16:34:07.8723345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8723543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8723607Z res = mod(**inputs) 2025-11-03T16:34:07.8723813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8723893Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8724182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8724256Z outputs = self.layoutlm( 2025-11-03T16:34:07.8724465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8724535Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8724801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8724870Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8725087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8725173Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8725433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8725502Z layer_outputs = layer_module( 2025-11-03T16:34:07.8725713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8725808Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8726054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8726140Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8726380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8726452Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8726739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8726855Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8727106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8727183Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8727186Z 2025-11-03T16:34:07.8727287Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8727471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8727532Z res = mod(**inputs) 2025-11-03T16:34:07.8727768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8727839Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8728096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8728162Z outputs = self.layoutlm( 2025-11-03T16:34:07.8728365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8728445Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8728692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8728768Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8728968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8729046Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8729298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8729365Z layer_outputs = layer_module( 2025-11-03T16:34:07.8729579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8729652Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8729927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8730006Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8730249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8730330Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8730613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8730734Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8730998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8731110Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8731311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8731379Z return self.act(input) 2025-11-03T16:34:07.8731382Z 2025-11-03T16:34:07.8731494Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8731681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8731751Z res = mod(**inputs) 2025-11-03T16:34:07.8731964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8732038Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8732295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8732365Z outputs = self.layoutlm( 2025-11-03T16:34:07.8732576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8732650Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8732903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8732982Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8733186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8733263Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8733543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8733615Z layer_outputs = layer_module( 2025-11-03T16:34:07.8733820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8733893Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8734146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8734224Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8734471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8734540Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8734825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8734955Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8735203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8735288Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8735291Z 2025-11-03T16:34:07.8735386Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8735592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8735653Z res = mod(**inputs) 2025-11-03T16:34:07.8735859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8735936Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8736191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8736267Z outputs = self.layoutlm( 2025-11-03T16:34:07.8736481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8736565Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8736822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8736892Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8737102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8737169Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8737417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8737490Z layer_outputs = layer_module( 2025-11-03T16:34:07.8737695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8737779Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8738028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8738115Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8738362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8738429Z self_outputs = self.self( 2025-11-03T16:34:07.8738682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8738818Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8738821Z 2025-11-03T16:34:07.8738925Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8739136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8739199Z res = mod(**inputs) 2025-11-03T16:34:07.8739410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8739481Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8739735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8739802Z outputs = self.layoutlm( 2025-11-03T16:34:07.8740012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8740080Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8740330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8740408Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8740610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8740685Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8740935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8741001Z layer_outputs = layer_module( 2025-11-03T16:34:07.8741228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8741301Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8741557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8741633Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8741881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8741956Z self_outputs = self.self( 2025-11-03T16:34:07.8742203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8742352Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8742356Z 2025-11-03T16:34:07.8742451Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8742646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8742707Z res = mod(**inputs) 2025-11-03T16:34:07.8742910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8742986Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8743230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8743304Z outputs = self.layoutlm( 2025-11-03T16:34:07.8743509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8743579Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8743832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8743902Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8744110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8744179Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8744434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8744499Z layer_outputs = layer_module( 2025-11-03T16:34:07.8744740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8744822Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8745067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8745152Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8745396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8745463Z self_outputs = self.self( 2025-11-03T16:34:07.8745716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8745851Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8745854Z 2025-11-03T16:34:07.8745936Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8746009Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8746105Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8746295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8746358Z res = mod(**inputs) 2025-11-03T16:34:07.8746566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8746634Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8746905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8746971Z outputs = self.layoutlm( 2025-11-03T16:34:07.8747171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8747247Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8747494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8747569Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8747769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8747852Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8748114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8748181Z layer_outputs = layer_module( 2025-11-03T16:34:07.8748397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8748470Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8748730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8748815Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8749072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8749205Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8749459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8749544Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8749548Z 2025-11-03T16:34:07.8749645Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8749834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8749903Z res = mod(**inputs) 2025-11-03T16:34:07.8750111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8750187Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8750473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8750541Z outputs = self.layoutlm( 2025-11-03T16:34:07.8750752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8750821Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8751079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8751149Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8751357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8751424Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8751672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8751747Z layer_outputs = layer_module( 2025-11-03T16:34:07.8751955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8752037Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8752286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8752384Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8752637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8752711Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8753009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8753132Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8753398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8753478Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8753497Z 2025-11-03T16:34:07.8753597Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8753792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8753856Z res = mod(**inputs) 2025-11-03T16:34:07.8754157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8754233Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8754498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8754578Z outputs = self.layoutlm( 2025-11-03T16:34:07.8754796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8754877Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8755140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8755221Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8755430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8755503Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8755764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8755832Z layer_outputs = layer_module( 2025-11-03T16:34:07.8756055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8756130Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8756424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8756518Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8756768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8756850Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8757136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8757250Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8757512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8757620Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8757835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8757905Z return self.act(input) 2025-11-03T16:34:07.8757908Z 2025-11-03T16:34:07.8758016Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8758207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8758271Z res = mod(**inputs) 2025-11-03T16:34:07.8758505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8758574Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8758842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8758909Z outputs = self.layoutlm( 2025-11-03T16:34:07.8759118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8759196Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8759455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8759544Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8759754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8759830Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8760088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8760157Z layer_outputs = layer_module( 2025-11-03T16:34:07.8760382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8760455Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8760719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8760797Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8761053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8761134Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8761421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8761559Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8761817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8761903Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8761906Z 2025-11-03T16:34:07.8762005Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8762229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8762301Z res = mod(**inputs) 2025-11-03T16:34:07.8762510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8762585Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8762843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8762912Z outputs = self.layoutlm( 2025-11-03T16:34:07.8763132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8763201Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8763470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8763539Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8763754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8763834Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8764098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8764175Z layer_outputs = layer_module( 2025-11-03T16:34:07.8764408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8764490Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8764746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8764826Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8765091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8765160Z self_outputs = self.self( 2025-11-03T16:34:07.8765424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8765579Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8765582Z 2025-11-03T16:34:07.8765680Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8765880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8765942Z res = mod(**inputs) 2025-11-03T16:34:07.8766159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8766227Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8766494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8766561Z outputs = self.layoutlm( 2025-11-03T16:34:07.8766767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8766846Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8767100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8767180Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8767390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8767460Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8767725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8767792Z layer_outputs = layer_module( 2025-11-03T16:34:07.8768053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8768131Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8768389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8768476Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8768738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8768815Z self_outputs = self.self( 2025-11-03T16:34:07.8769070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8769206Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8769210Z 2025-11-03T16:34:07.8769307Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8769504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8769575Z res = mod(**inputs) 2025-11-03T16:34:07.8769781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8769858Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8770112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8770196Z outputs = self.layoutlm( 2025-11-03T16:34:07.8770423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8770492Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8770758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8770828Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8771047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8771119Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8771397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8771474Z layer_outputs = layer_module( 2025-11-03T16:34:07.8771689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8771770Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8772025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8772102Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8772370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8772436Z self_outputs = self.self( 2025-11-03T16:34:07.8772702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8772840Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8772844Z 2025-11-03T16:34:07.8772930Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8773010Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8773110Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8773308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8773370Z res = mod(**inputs) 2025-11-03T16:34:07.8773595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8773667Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8773956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8774035Z outputs = self.layoutlm( 2025-11-03T16:34:07.8774244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8774321Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8774579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8774648Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8774872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8774941Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8775203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8775274Z layer_outputs = layer_module( 2025-11-03T16:34:07.8775486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8775569Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8775824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8775928Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8776186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8776315Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8776568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8776646Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8776652Z 2025-11-03T16:34:07.8776759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8776943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8777026Z res = mod(**inputs) 2025-11-03T16:34:07.8777236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8777305Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8777571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8777641Z outputs = self.layoutlm( 2025-11-03T16:34:07.8777857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8777926Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8778195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8778264Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8778473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8778551Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8778809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8778885Z layer_outputs = layer_module( 2025-11-03T16:34:07.8779098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8779172Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8779440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8779519Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8779811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8779891Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8780182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8780303Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8780558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8780645Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8780648Z 2025-11-03T16:34:07.8780746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8780941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8781003Z res = mod(**inputs) 2025-11-03T16:34:07.8781215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8781297Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8781550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8781626Z outputs = self.layoutlm( 2025-11-03T16:34:07.8781846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8781919Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8782182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8782249Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8782463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8782536Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8782798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8782894Z layer_outputs = layer_module( 2025-11-03T16:34:07.8783113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8783199Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8783468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8783564Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8783815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8783888Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8784184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8784299Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8784562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8784672Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8784891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8784959Z return self.act(input) 2025-11-03T16:34:07.8784963Z 2025-11-03T16:34:07.8785061Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8785258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8785319Z res = mod(**inputs) 2025-11-03T16:34:07.8785564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8785636Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8785892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8785967Z outputs = self.layoutlm( 2025-11-03T16:34:07.8786173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8786252Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8786509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8786586Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8786791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8786862Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8787127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8787195Z layer_outputs = layer_module( 2025-11-03T16:34:07.8787415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8787491Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8787762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8787848Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8788091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8788168Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8788447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8788569Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8788846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8788922Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8788925Z 2025-11-03T16:34:07.8789031Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8789211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8789280Z res = mod(**inputs) 2025-11-03T16:34:07.8789481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8789550Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8789806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8789871Z outputs = self.layoutlm( 2025-11-03T16:34:07.8790078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8790148Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8790401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8790477Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8790689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8790763Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8791018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8791094Z layer_outputs = layer_module( 2025-11-03T16:34:07.8791332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8791407Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8791671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8791751Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8792011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8792079Z self_outputs = self.self( 2025-11-03T16:34:07.8792334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8792483Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8792486Z 2025-11-03T16:34:07.8792583Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8792780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8792842Z res = mod(**inputs) 2025-11-03T16:34:07.8793057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8793127Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8793381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8793472Z outputs = self.layoutlm( 2025-11-03T16:34:07.8793681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8793758Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8794081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8794166Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8794388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8794484Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8794746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8794814Z layer_outputs = layer_module( 2025-11-03T16:34:07.8795040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8795125Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8795387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8795476Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8795729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8795806Z self_outputs = self.self( 2025-11-03T16:34:07.8796054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8796186Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8796190Z 2025-11-03T16:34:07.8796298Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8796484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8796554Z res = mod(**inputs) 2025-11-03T16:34:07.8796756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8796824Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8797113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8797180Z outputs = self.layoutlm( 2025-11-03T16:34:07.8797390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8797459Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8797715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8797784Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8797982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8798059Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8798302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8798377Z layer_outputs = layer_module( 2025-11-03T16:34:07.8798586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8798661Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8798915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8798989Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8799241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8799325Z self_outputs = self.self( 2025-11-03T16:34:07.8799574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8799716Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8799719Z 2025-11-03T16:34:07.8799793Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8799878Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8799976Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8800163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8800240Z res = mod(**inputs) 2025-11-03T16:34:07.8800440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8800518Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8800763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8800833Z outputs = self.layoutlm( 2025-11-03T16:34:07.8801032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8801099Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8801353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8801421Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8801629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8801696Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8801941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8802017Z layer_outputs = layer_module( 2025-11-03T16:34:07.8802221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8802301Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8802547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8802660Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8802909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8803030Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8803288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8803367Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8803370Z 2025-11-03T16:34:07.8803473Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8803660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8803721Z res = mod(**inputs) 2025-11-03T16:34:07.8803929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8803997Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8804252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8804319Z outputs = self.layoutlm( 2025-11-03T16:34:07.8804530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8804599Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8804861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8804935Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8805137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8805217Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8805461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8805530Z layer_outputs = layer_module( 2025-11-03T16:34:07.8805745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8805834Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8806090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8806171Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8806415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8806495Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8806772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8806896Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8807145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8807231Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8807235Z 2025-11-03T16:34:07.8807332Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8807517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8807589Z res = mod(**inputs) 2025-11-03T16:34:07.8807791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8807869Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8808118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8808185Z outputs = self.layoutlm( 2025-11-03T16:34:07.8808449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8808520Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8808779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8808847Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8809059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8809130Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8809387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8809461Z layer_outputs = layer_module( 2025-11-03T16:34:07.8809674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8809769Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8810023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8810105Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8810365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8810437Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8810752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8810863Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8811118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8811222Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8811424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8811499Z return self.act(input) 2025-11-03T16:34:07.8811517Z 2025-11-03T16:34:07.8811614Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8811807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8811869Z res = mod(**inputs) 2025-11-03T16:34:07.8812077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8812153Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8812403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8812477Z outputs = self.layoutlm( 2025-11-03T16:34:07.8812680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8812753Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8813008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8813079Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8813446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8813523Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8813780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8813847Z layer_outputs = layer_module( 2025-11-03T16:34:07.8814053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8814137Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8814442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8814530Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8814777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8814847Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8815131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8815260Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8815521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8815598Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8815601Z 2025-11-03T16:34:07.8815705Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8815891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8815953Z res = mod(**inputs) 2025-11-03T16:34:07.8816165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8816233Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8816494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8816587Z outputs = self.layoutlm( 2025-11-03T16:34:07.8816790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8816869Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8817119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8817199Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8817401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8817501Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8817761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8817827Z layer_outputs = layer_module( 2025-11-03T16:34:07.8818050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8818123Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8818379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8818457Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8818707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8818780Z self_outputs = self.self( 2025-11-03T16:34:07.8819031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8819174Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8819178Z 2025-11-03T16:34:07.8819273Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8819463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8819524Z res = mod(**inputs) 2025-11-03T16:34:07.8819727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8819803Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8820080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8820154Z outputs = self.layoutlm( 2025-11-03T16:34:07.8820356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8820426Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8820691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8820760Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8820966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8821037Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8821286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8821360Z layer_outputs = layer_module( 2025-11-03T16:34:07.8821567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8821649Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8821897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8821980Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8822226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8822311Z self_outputs = self.self( 2025-11-03T16:34:07.8822569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8822699Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8822702Z 2025-11-03T16:34:07.8822805Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8822991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8823055Z res = mod(**inputs) 2025-11-03T16:34:07.8823281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8823349Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8823604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8823671Z outputs = self.layoutlm( 2025-11-03T16:34:07.8823883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8823953Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8824202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8824279Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8824482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8824558Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8824808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8824876Z layer_outputs = layer_module( 2025-11-03T16:34:07.8825094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8825166Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8825421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8825497Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8825776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8825851Z self_outputs = self.self( 2025-11-03T16:34:07.8826099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8826244Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8826247Z 2025-11-03T16:34:07.8826327Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8826412Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8826511Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8826699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8826768Z res = mod(**inputs) 2025-11-03T16:34:07.8826974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8827052Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8827303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8827373Z outputs = self.layoutlm( 2025-11-03T16:34:07.8827586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8827657Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8827930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8827999Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8828201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8828275Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8828528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8828601Z layer_outputs = layer_module( 2025-11-03T16:34:07.8828807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8845968Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8846432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8846543Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8846818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8846956Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8847218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8847320Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8847328Z 2025-11-03T16:34:07.8847438Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8847656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8847728Z res = mod(**inputs) 2025-11-03T16:34:07.8847947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8848034Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8848288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8848371Z outputs = self.layoutlm( 2025-11-03T16:34:07.8848581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8848654Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8849050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8849129Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8849344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8849415Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8849663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8849744Z layer_outputs = layer_module( 2025-11-03T16:34:07.8849958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8850048Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8850299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8850393Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8850640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8850717Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8851006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8851150Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8851408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8851488Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8851492Z 2025-11-03T16:34:07.8851596Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8851798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8851866Z res = mod(**inputs) 2025-11-03T16:34:07.8852081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8852178Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8852435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8852507Z outputs = self.layoutlm( 2025-11-03T16:34:07.8852715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8852794Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8853051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8853131Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8853338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8853407Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8853666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8853738Z layer_outputs = layer_module( 2025-11-03T16:34:07.8853956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8854032Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8854279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8854364Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8854607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8854690Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8854998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8855123Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8855376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8855485Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8855699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8855768Z return self.act(input) 2025-11-03T16:34:07.8855773Z 2025-11-03T16:34:07.8855880Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8856069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8856132Z res = mod(**inputs) 2025-11-03T16:34:07.8856347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8856417Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8856678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8856746Z outputs = self.layoutlm( 2025-11-03T16:34:07.8856957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8857047Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8857301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8857381Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8857587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8857666Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8857917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8858004Z layer_outputs = layer_module( 2025-11-03T16:34:07.8858217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8858291Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8858546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8858623Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8858868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8858940Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8859217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8859352Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8859601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8859687Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8859692Z 2025-11-03T16:34:07.8859789Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8859975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8860045Z res = mod(**inputs) 2025-11-03T16:34:07.8860253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8860330Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8860612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8860690Z outputs = self.layoutlm( 2025-11-03T16:34:07.8860893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8860964Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8861224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8861296Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8861507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8861576Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8861827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8861903Z layer_outputs = layer_module( 2025-11-03T16:34:07.8862114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8862199Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8862453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8862533Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8862806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8862876Z self_outputs = self.self( 2025-11-03T16:34:07.8863130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-11-03T16:34:07.8863270Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8863274Z 2025-11-03T16:34:07.8863382Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8863566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8863643Z res = mod(**inputs) 2025-11-03T16:34:07.8863861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8863933Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8864195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8864266Z outputs = self.layoutlm( 2025-11-03T16:34:07.8864473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8864553Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8864804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8864887Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8865095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8865177Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8865427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8865500Z layer_outputs = layer_module( 2025-11-03T16:34:07.8865718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8865795Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8866055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8866138Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8866417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8866497Z self_outputs = self.self( 2025-11-03T16:34:07.8866750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-11-03T16:34:07.8866890Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8866897Z 2025-11-03T16:34:07.8866996Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8867186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8867248Z res = mod(**inputs) 2025-11-03T16:34:07.8867452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8867529Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8867781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8867854Z outputs = self.layoutlm( 2025-11-03T16:34:07.8868057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8868129Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8868389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8868471Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8868689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8868759Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8869013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8869086Z layer_outputs = layer_module( 2025-11-03T16:34:07.8869294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8869377Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8869643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8869728Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8869976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 258, in forward 2025-11-03T16:34:07.8870041Z self_outputs = self.self( 2025-11-03T16:34:07.8870318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 186, in forward 2025-11-03T16:34:07.8870453Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-11-03T16:34:07.8870456Z 2025-11-03T16:34:07.8870543Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8870622Z cudagraph partition due to non gpu ops 2025-11-03T16:34:07.8870720Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8870913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8870975Z res = mod(**inputs) 2025-11-03T16:34:07.8871184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8871255Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8871500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8871573Z outputs = self.layoutlm( 2025-11-03T16:34:07.8871776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8871852Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8872140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8872218Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8872424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8872492Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8872749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8872818Z layer_outputs = layer_module( 2025-11-03T16:34:07.8873028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8873103Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8873349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 319, in forward 2025-11-03T16:34:07.8873436Z self_attention_outputs = self.attention( 2025-11-03T16:34:07.8873728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 265, in forward 2025-11-03T16:34:07.8873868Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:34:07.8874230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 218, in forward 2025-11-03T16:34:07.8874352Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8874356Z 2025-11-03T16:34:07.8874459Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8874684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8874761Z res = mod(**inputs) 2025-11-03T16:34:07.8874980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8875067Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8875334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8875426Z outputs = self.layoutlm( 2025-11-03T16:34:07.8875655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8875729Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8876003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8876078Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8876291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8876374Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8876643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8876721Z layer_outputs = layer_module( 2025-11-03T16:34:07.8876936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8877026Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8877288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8877373Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8877646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8877723Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8878024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8878175Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8878438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-11-03T16:34:07.8878529Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8878533Z 2025-11-03T16:34:07.8878635Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8878834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8878901Z res = mod(**inputs) 2025-11-03T16:34:07.8879122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8879195Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8879455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8879536Z outputs = self.layoutlm( 2025-11-03T16:34:07.8879754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8879836Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8880100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8880173Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8880397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8880486Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8880756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8880828Z layer_outputs = layer_module( 2025-11-03T16:34:07.8881045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8881135Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8881399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8881508Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8881763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8881849Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8882142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-11-03T16:34:07.8882262Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:34:07.8882536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 282, in forward 2025-11-03T16:34:07.8882648Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:34:07.8882897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:34:07.8882967Z return self.act(input) 2025-11-03T16:34:07.8882972Z 2025-11-03T16:34:07.8883073Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8883277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8883343Z res = mod(**inputs) 2025-11-03T16:34:07.8883566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8883639Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8883912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 722, in forward 2025-11-03T16:34:07.8883980Z outputs = self.layoutlm( 2025-11-03T16:34:07.8884229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8884311Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8884575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 614, in forward 2025-11-03T16:34:07.8884658Z encoder_outputs = self.encoder( 2025-11-03T16:34:07.8884874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8884948Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8885220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 370, in forward 2025-11-03T16:34:07.8885290Z layer_outputs = layer_module( 2025-11-03T16:34:07.8885516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:07.8885594Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:07.8885862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 329, in forward 2025-11-03T16:34:07.8885952Z layer_output = apply_chunking_to_forward( 2025-11-03T16:34:07.8886209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:34:07.8886292Z return forward_fn(*input_tensors) 2025-11-03T16:34:07.8886604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in feed_forward_chunk 2025-11-03T16:34:07.8886746Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:34:07.8887009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-11-03T16:34:07.8887087Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8887090Z 2025-11-03T16:34:07.8887203Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8887398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8887485Z res = mod(**inputs) 2025-11-03T16:34:07.8887710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8887780Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8888041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 736, in forward 2025-11-03T16:34:07.8888131Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:34:07.8888393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 457, in forward 2025-11-03T16:34:07.8888499Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:34:07.8888764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 445, in forward 2025-11-03T16:34:07.8888853Z hidden_states = self.transform(hidden_states) 2025-11-03T16:34:07.8889108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 420, in forward 2025-11-03T16:34:07.8889196Z hidden_states = self.dense(hidden_states) 2025-11-03T16:34:07.8889201Z 2025-11-03T16:34:07.8889297Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8889494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8889555Z res = mod(**inputs) 2025-11-03T16:34:07.8889761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8889843Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8890124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 736, in forward 2025-11-03T16:34:07.8890218Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:34:07.8890467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 457, in forward 2025-11-03T16:34:07.8890578Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:34:07.8890828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 446, in forward 2025-11-03T16:34:07.8890916Z hidden_states = self.decoder(hidden_states) 2025-11-03T16:34:07.8890919Z 2025-11-03T16:34:07.8891024Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:07.8891205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:07.8891273Z res = mod(**inputs) 2025-11-03T16:34:07.8891475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:34:07.8891547Z output = func(self, *args, **kwargs) 2025-11-03T16:34:07.8891801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 741, in forward 2025-11-03T16:34:07.8891873Z masked_lm_loss = loss_fct( 2025-11-03T16:34:07.8891876Z 2025-11-03T16:34:17.3214228Z Compilation time (from dynamo_timed): 16.437832384 2025-11-03T16:34:17.3265214Z pass 2025-11-03T16:34:17.3267065Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:17.3267847Z TIMING: _recursive_pre_grad_passes:0.00729 _recursive_joint_graph_passes:0.72714 _recursive_post_grad_passes:0.06262 async_compile.wait:0.59435 code_gen:8.68843 inductor_compile:9.96723 backend_compile:13.40761 gc:0.00014 entire_frame_compile:16.43783 total_wall_time:16.43783 2025-11-03T16:34:17.3268849Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:8741 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:2595 2025-11-03T16:34:17.3269333Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-11-03T16:34:19.5281022Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:34:19.5282222Z import pynvml # type: ignore[import] 2025-11-03T16:34:22.7091784Z 2025-11-03T16:34:28.2293146Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:34:28.2297807Z loading model: 0it [00:05, ?it/s] 2025-11-03T16:34:28.2316108Z cpu eval M2M100ForConditionalGeneration 2025-11-03T16:34:28.9857057Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:29.2984537Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:29.6183415Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:44.8514737Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8515223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8515568Z res = mod(**inputs) 2025-11-03T16:34:44.8516046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8516456Z outputs = self.model( 2025-11-03T16:34:44.8516825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8517219Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8517606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 848, in forward 2025-11-03T16:34:44.8518456Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-11-03T16:34:44.8518874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-11-03T16:34:44.8519247Z return func(*args, **kwargs) 2025-11-03T16:34:44.8519620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-11-03T16:34:44.8520149Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-11-03T16:34:44.8520726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-11-03T16:34:44.8521181Z mask = input_ids.ne(padding_idx).int() 2025-11-03T16:34:44.8521326Z 2025-11-03T16:34:44.8521442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8521809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8522132Z res = mod(**inputs) 2025-11-03T16:34:44.8522499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8522881Z outputs = self.model( 2025-11-03T16:34:44.8523281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.8523748Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.8524129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1099, in forward 2025-11-03T16:34:44.8524600Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-11-03T16:34:44.8525071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-11-03T16:34:44.8525446Z return func(*args, **kwargs) 2025-11-03T16:34:44.8525858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-11-03T16:34:44.8527051Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-11-03T16:34:44.8527669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-11-03T16:34:44.8528122Z mask = input_ids.ne(padding_idx).int() 2025-11-03T16:34:44.8528261Z 2025-11-03T16:34:44.8528352Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8528561Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8528771Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8528974Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8529178Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8529372Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8529577Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8529777Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8529990Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8530186Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8530373Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8530565Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8530789Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8531140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8531459Z res = mod(**inputs) 2025-11-03T16:34:44.8531835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8532198Z outputs = self.model( 2025-11-03T16:34:44.8532545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8532939Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8533297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 848, in forward 2025-11-03T16:34:44.8533698Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-11-03T16:34:44.8534081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-11-03T16:34:44.8534430Z return func(*args, **kwargs) 2025-11-03T16:34:44.8534781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-11-03T16:34:44.8535274Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-11-03T16:34:44.8535834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-11-03T16:34:44.8536364Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-11-03T16:34:44.8536586Z 2025-11-03T16:34:44.8536693Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8537024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8537335Z res = mod(**inputs) 2025-11-03T16:34:44.8537685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8538075Z outputs = self.model( 2025-11-03T16:34:44.8538424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8538792Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8539159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 848, in forward 2025-11-03T16:34:44.8539572Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-11-03T16:34:44.8539951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-11-03T16:34:44.8540316Z return func(*args, **kwargs) 2025-11-03T16:34:44.8540670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-11-03T16:34:44.8541161Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-11-03T16:34:44.8541704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-11-03T16:34:44.8542239Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-11-03T16:34:44.8542465Z 2025-11-03T16:34:44.8542573Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8542915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8543227Z res = mod(**inputs) 2025-11-03T16:34:44.8543584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8543958Z outputs = self.model( 2025-11-03T16:34:44.8544306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8544680Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8545046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8545417Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8545766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8546149Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8546523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8546912Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8547296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8547658Z return func(*args, **kwargs) 2025-11-03T16:34:44.8548009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8548453Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8548655Z 2025-11-03T16:34:44.8548755Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8549097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8549411Z res = mod(**inputs) 2025-11-03T16:34:44.8549756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8550130Z outputs = self.model( 2025-11-03T16:34:44.8550484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8550877Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8551239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8551610Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8551947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8552300Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8552778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8553159Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8553556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8553936Z return func(*args, **kwargs) 2025-11-03T16:34:44.8554439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8554846Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8554986Z 2025-11-03T16:34:44.8555100Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8555455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8555781Z res = mod(**inputs) 2025-11-03T16:34:44.8556157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8556532Z outputs = self.model( 2025-11-03T16:34:44.8556903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8557277Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8557655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8558038Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8558377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8558722Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8559142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8559537Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8559970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8560332Z return func(*args, **kwargs) 2025-11-03T16:34:44.8560684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8561078Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8561223Z 2025-11-03T16:34:44.8561304Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8561545Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8561888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8562242Z res = mod(**inputs) 2025-11-03T16:34:44.8562604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8562992Z outputs = self.model( 2025-11-03T16:34:44.8563359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8563734Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8564121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8564489Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8564859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8565228Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8565616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8566026Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8566426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8566831Z return func(*args, **kwargs) 2025-11-03T16:34:44.8567205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8567636Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8568086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8568567Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8568748Z 2025-11-03T16:34:44.8568856Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8569204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8569522Z res = mod(**inputs) 2025-11-03T16:34:44.8569884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8570273Z outputs = self.model( 2025-11-03T16:34:44.8570646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8571031Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8571424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8571850Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8572209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8572593Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8573008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8573423Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8573873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8574254Z return func(*args, **kwargs) 2025-11-03T16:34:44.8574625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8575021Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8575167Z 2025-11-03T16:34:44.8575272Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8575633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8575957Z res = mod(**inputs) 2025-11-03T16:34:44.8576315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8576701Z outputs = self.model( 2025-11-03T16:34:44.8577069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8577460Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8577843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8578224Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8578575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8578955Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8579343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8579774Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8579956Z 2025-11-03T16:34:44.8580059Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8580420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8580744Z res = mod(**inputs) 2025-11-03T16:34:44.8581131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8581558Z outputs = self.model( 2025-11-03T16:34:44.8581952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8582369Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8582776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8583200Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8583569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8583957Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8584368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8584807Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8584983Z 2025-11-03T16:34:44.8585095Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8585446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8585778Z res = mod(**inputs) 2025-11-03T16:34:44.8586145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8586536Z outputs = self.model( 2025-11-03T16:34:44.8586900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8587298Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8587713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8588101Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8588452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8588810Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8589211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8589612Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8589752Z 2025-11-03T16:34:44.8589869Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8590226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8590541Z res = mod(**inputs) 2025-11-03T16:34:44.8590908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8591287Z outputs = self.model( 2025-11-03T16:34:44.8591652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8592034Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8592416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8592822Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8593186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8593571Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8594073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8594533Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8595010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8595415Z return func(*args, **kwargs) 2025-11-03T16:34:44.8595808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8596273Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8596488Z 2025-11-03T16:34:44.8596594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8596956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8597283Z res = mod(**inputs) 2025-11-03T16:34:44.8597628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8597996Z outputs = self.model( 2025-11-03T16:34:44.8598350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8598721Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8599085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8599447Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8599781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8600130Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8600501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8600885Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8601254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8601642Z return func(*args, **kwargs) 2025-11-03T16:34:44.8602006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8602383Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8602513Z 2025-11-03T16:34:44.8602612Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8602954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8603263Z res = mod(**inputs) 2025-11-03T16:34:44.8603610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8603980Z outputs = self.model( 2025-11-03T16:34:44.8604321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8604697Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8605066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8605437Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8605763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8606107Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8606499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8606896Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8607291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8607647Z return func(*args, **kwargs) 2025-11-03T16:34:44.8608013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8608394Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8608531Z 2025-11-03T16:34:44.8608618Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8608863Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8609198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8609506Z res = mod(**inputs) 2025-11-03T16:34:44.8609861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8610227Z outputs = self.model( 2025-11-03T16:34:44.8610569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8610937Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8611303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8611668Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8611999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8612346Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8612718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8613104Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8613681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8614048Z return func(*args, **kwargs) 2025-11-03T16:34:44.8614402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8614800Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8615322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8615826Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8616012Z 2025-11-03T16:34:44.8616130Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8616482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8616804Z res = mod(**inputs) 2025-11-03T16:34:44.8617162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8617543Z outputs = self.model( 2025-11-03T16:34:44.8617894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8618275Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8618654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8619032Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8619382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8619733Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8620117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8620550Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8620939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8621311Z return func(*args, **kwargs) 2025-11-03T16:34:44.8621672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8622063Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8622205Z 2025-11-03T16:34:44.8622307Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8622687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8622995Z res = mod(**inputs) 2025-11-03T16:34:44.8623352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8623726Z outputs = self.model( 2025-11-03T16:34:44.8624082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8624462Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8624826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8625218Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8625578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8625934Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8626315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8626735Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8626916Z 2025-11-03T16:34:44.8627018Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8627364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8627675Z res = mod(**inputs) 2025-11-03T16:34:44.8628022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8628403Z outputs = self.model( 2025-11-03T16:34:44.8628800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8629187Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8629572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8629953Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8630310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8630682Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8631078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8631515Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8631686Z 2025-11-03T16:34:44.8631791Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8632152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8632477Z res = mod(**inputs) 2025-11-03T16:34:44.8632853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8633235Z outputs = self.model( 2025-11-03T16:34:44.8633605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8634086Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8634479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8634881Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8635253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8635614Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8636002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8636420Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8636559Z 2025-11-03T16:34:44.8636672Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8637024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8637349Z res = mod(**inputs) 2025-11-03T16:34:44.8637719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8638102Z outputs = self.model( 2025-11-03T16:34:44.8638462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8638848Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8639230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8639618Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8639970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8640333Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8640723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-11-03T16:34:44.8641114Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.8641251Z 2025-11-03T16:34:44.8641361Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8641719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8642037Z res = mod(**inputs) 2025-11-03T16:34:44.8642432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8642815Z outputs = self.model( 2025-11-03T16:34:44.8643184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8643566Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8643947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8644335Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8644680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8645021Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8645380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8645769Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8646145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8646505Z return func(*args, **kwargs) 2025-11-03T16:34:44.8646864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8647301Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8647525Z 2025-11-03T16:34:44.8647623Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8647963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8648271Z res = mod(**inputs) 2025-11-03T16:34:44.8648613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8648985Z outputs = self.model( 2025-11-03T16:34:44.8649345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8649730Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8650091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8650451Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8650790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8651136Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8651511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8651895Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8652267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8652624Z return func(*args, **kwargs) 2025-11-03T16:34:44.8652983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8653354Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8653484Z 2025-11-03T16:34:44.8653580Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8653924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8654230Z res = mod(**inputs) 2025-11-03T16:34:44.8654579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8654944Z outputs = self.model( 2025-11-03T16:34:44.8655287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8655693Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8656060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8656433Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8656780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8657156Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8657530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8657916Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8658295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8658663Z return func(*args, **kwargs) 2025-11-03T16:34:44.8659038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8659442Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8659584Z 2025-11-03T16:34:44.8659671Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8659905Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8660248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8660586Z res = mod(**inputs) 2025-11-03T16:34:44.8660951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8661403Z outputs = self.model( 2025-11-03T16:34:44.8661770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8662150Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8662533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8662915Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8663288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8663643Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8664026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8664426Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8664816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8665187Z return func(*args, **kwargs) 2025-11-03T16:34:44.8665548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8665956Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8666401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8666879Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8667061Z 2025-11-03T16:34:44.8667167Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8667522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8667841Z res = mod(**inputs) 2025-11-03T16:34:44.8668203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8668583Z outputs = self.model( 2025-11-03T16:34:44.8668938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8669351Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8669727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8670105Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8670449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8670803Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8671190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8671587Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8671970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8672338Z return func(*args, **kwargs) 2025-11-03T16:34:44.8672699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8673088Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8673231Z 2025-11-03T16:34:44.8673335Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8673683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8674069Z res = mod(**inputs) 2025-11-03T16:34:44.8674459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8674875Z outputs = self.model( 2025-11-03T16:34:44.8675277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8675682Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8676066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8676465Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8676824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8677217Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8677608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8678036Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8678218Z 2025-11-03T16:34:44.8678322Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8678682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8679007Z res = mod(**inputs) 2025-11-03T16:34:44.8679375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8679755Z outputs = self.model( 2025-11-03T16:34:44.8680133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8680545Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8680939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8681322Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8681672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8682048Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8682458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8682913Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8683095Z 2025-11-03T16:34:44.8683236Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8683616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8683959Z res = mod(**inputs) 2025-11-03T16:34:44.8684349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8684730Z outputs = self.model( 2025-11-03T16:34:44.8685094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8685495Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8685880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8686342Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8686689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8687053Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8687445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8687843Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8687978Z 2025-11-03T16:34:44.8688090Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8688456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8688779Z res = mod(**inputs) 2025-11-03T16:34:44.8689149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8689533Z outputs = self.model( 2025-11-03T16:34:44.8689895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8690291Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8690672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8691083Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8691438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8691797Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8692198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8692608Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8693007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8693386Z return func(*args, **kwargs) 2025-11-03T16:34:44.8693762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8694228Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8694442Z 2025-11-03T16:34:44.8694548Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8694920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8695228Z res = mod(**inputs) 2025-11-03T16:34:44.8695569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8695938Z outputs = self.model( 2025-11-03T16:34:44.8696292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8696671Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8697068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8697444Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8697776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8698125Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8698494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8698870Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8699246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8699607Z return func(*args, **kwargs) 2025-11-03T16:34:44.8699962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8700336Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8700467Z 2025-11-03T16:34:44.8700567Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8700905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8701216Z res = mod(**inputs) 2025-11-03T16:34:44.8701561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8701938Z outputs = self.model( 2025-11-03T16:34:44.8702288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8702656Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8703023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8703390Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8703726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8704072Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8704525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8704920Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8705300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8705655Z return func(*args, **kwargs) 2025-11-03T16:34:44.8706018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8706409Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8706545Z 2025-11-03T16:34:44.8706634Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8706859Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8707201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8707512Z res = mod(**inputs) 2025-11-03T16:34:44.8707867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8708241Z outputs = self.model( 2025-11-03T16:34:44.8708584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8708958Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8709320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8709690Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8710015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8710394Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8710766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8711159Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8711544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8711905Z return func(*args, **kwargs) 2025-11-03T16:34:44.8712267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8712669Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8713112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8713749Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8713935Z 2025-11-03T16:34:44.8714084Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8714464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8714798Z res = mod(**inputs) 2025-11-03T16:34:44.8715174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8715603Z outputs = self.model( 2025-11-03T16:34:44.8715956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8716333Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8716705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8717082Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8717422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8717780Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8718186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8718583Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8718969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8719335Z return func(*args, **kwargs) 2025-11-03T16:34:44.8719704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8720090Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8720225Z 2025-11-03T16:34:44.8720332Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8720680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8720986Z res = mod(**inputs) 2025-11-03T16:34:44.8721340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8721717Z outputs = self.model( 2025-11-03T16:34:44.8722074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8722450Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8722823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8723195Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8723535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8723890Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8724310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8724737Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8724923Z 2025-11-03T16:34:44.8725022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8725351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8725654Z res = mod(**inputs) 2025-11-03T16:34:44.8725986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8726347Z outputs = self.model( 2025-11-03T16:34:44.8726689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8727053Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8727410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8727780Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8728116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8728463Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8728836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8729264Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8729431Z 2025-11-03T16:34:44.8729530Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8729867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8730181Z res = mod(**inputs) 2025-11-03T16:34:44.8730517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8730866Z outputs = self.model( 2025-11-03T16:34:44.8731204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8731589Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8731957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8732308Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8732642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8732987Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8733355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8733728Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8733862Z 2025-11-03T16:34:44.8733962Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8734304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8734605Z res = mod(**inputs) 2025-11-03T16:34:44.8734941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8735305Z outputs = self.model( 2025-11-03T16:34:44.8735643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8736014Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8736370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8736725Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8737080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8737438Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8737808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-11-03T16:34:44.8738179Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.8738307Z 2025-11-03T16:34:44.8738412Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8738742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8739054Z res = mod(**inputs) 2025-11-03T16:34:44.8739402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8739769Z outputs = self.model( 2025-11-03T16:34:44.8740112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8740493Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8740848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8741207Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8741540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8741904Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8742279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8742667Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8743044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8743408Z return func(*args, **kwargs) 2025-11-03T16:34:44.8743770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8744222Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8744422Z 2025-11-03T16:34:44.8744522Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8744865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8745172Z res = mod(**inputs) 2025-11-03T16:34:44.8745510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8745877Z outputs = self.model( 2025-11-03T16:34:44.8746221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8746590Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8746946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8747315Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8747652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8747998Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8748367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8748749Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8749126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8749483Z return func(*args, **kwargs) 2025-11-03T16:34:44.8749842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8750249Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8750379Z 2025-11-03T16:34:44.8750478Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8750828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8751133Z res = mod(**inputs) 2025-11-03T16:34:44.8751479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8751839Z outputs = self.model( 2025-11-03T16:34:44.8752186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8752555Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8752916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8753284Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8753610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8753972Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8754481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8754963Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8755377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8755748Z return func(*args, **kwargs) 2025-11-03T16:34:44.8756110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8756493Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8756630Z 2025-11-03T16:34:44.8756718Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8756945Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8757292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8757622Z res = mod(**inputs) 2025-11-03T16:34:44.8757972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8758342Z outputs = self.model( 2025-11-03T16:34:44.8758698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8759071Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8759435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8759803Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8760134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8760484Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8760853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8761248Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8761610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8761954Z return func(*args, **kwargs) 2025-11-03T16:34:44.8762304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8762684Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8763101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8763642Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8763815Z 2025-11-03T16:34:44.8763911Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8764244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8764546Z res = mod(**inputs) 2025-11-03T16:34:44.8764898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8765258Z outputs = self.model( 2025-11-03T16:34:44.8765592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8765954Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8766314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8766685Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8767018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8767371Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8767745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8768131Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8768529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8768885Z return func(*args, **kwargs) 2025-11-03T16:34:44.8769242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8769615Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8769744Z 2025-11-03T16:34:44.8769849Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8770189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8770488Z res = mod(**inputs) 2025-11-03T16:34:44.8770854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8771216Z outputs = self.model( 2025-11-03T16:34:44.8771564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8771925Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8772286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8772654Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8772989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8773340Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8773703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8774114Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8774285Z 2025-11-03T16:34:44.8774382Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8774718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8775023Z res = mod(**inputs) 2025-11-03T16:34:44.8775361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8776396Z outputs = self.model( 2025-11-03T16:34:44.8776745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8777111Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8777511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8777884Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8778216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8778560Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8778931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8779336Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8779504Z 2025-11-03T16:34:44.8779605Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8779944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8780254Z res = mod(**inputs) 2025-11-03T16:34:44.8780614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8780974Z outputs = self.model( 2025-11-03T16:34:44.8781327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8781695Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8782058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8782439Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8782776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8783134Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8783519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8783907Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8784041Z 2025-11-03T16:34:44.8784153Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8784515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8784822Z res = mod(**inputs) 2025-11-03T16:34:44.8785168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8785540Z outputs = self.model( 2025-11-03T16:34:44.8785880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8786255Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8786620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8786987Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8787316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8787668Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8788044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8788429Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8788809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8789163Z return func(*args, **kwargs) 2025-11-03T16:34:44.8789524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8789964Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8790160Z 2025-11-03T16:34:44.8790301Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8790643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8790946Z res = mod(**inputs) 2025-11-03T16:34:44.8791291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8791655Z outputs = self.model( 2025-11-03T16:34:44.8792002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8792361Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8792722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8793084Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8793424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8793771Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8794228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8794635Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8795033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8795418Z return func(*args, **kwargs) 2025-11-03T16:34:44.8795777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8796143Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8796282Z 2025-11-03T16:34:44.8796380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8796730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8797055Z res = mod(**inputs) 2025-11-03T16:34:44.8797395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8797783Z outputs = self.model( 2025-11-03T16:34:44.8798122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8798489Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8798840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8799189Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8799514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8799848Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8800215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8800587Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8800943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8801292Z return func(*args, **kwargs) 2025-11-03T16:34:44.8801639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8802014Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8802144Z 2025-11-03T16:34:44.8802220Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8802442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8802775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8803074Z res = mod(**inputs) 2025-11-03T16:34:44.8803448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8803803Z outputs = self.model( 2025-11-03T16:34:44.8804154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8804524Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8804887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8805248Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8805582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8805929Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8806298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8806681Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8807047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8807407Z return func(*args, **kwargs) 2025-11-03T16:34:44.8807767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8808159Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8808589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8809029Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8809207Z 2025-11-03T16:34:44.8809304Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8809636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8809938Z res = mod(**inputs) 2025-11-03T16:34:44.8810275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8810646Z outputs = self.model( 2025-11-03T16:34:44.8810997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8811365Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8811733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8812092Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8812432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8812779Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8813153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8813649Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8814022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8814384Z return func(*args, **kwargs) 2025-11-03T16:34:44.8814744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8817556Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8817699Z 2025-11-03T16:34:44.8817800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8818151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8818459Z res = mod(**inputs) 2025-11-03T16:34:44.8818850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8819206Z outputs = self.model( 2025-11-03T16:34:44.8819546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8819908Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8820263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8820618Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8820969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8821305Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8821659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8822061Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8822229Z 2025-11-03T16:34:44.8822329Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8822664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8822966Z res = mod(**inputs) 2025-11-03T16:34:44.8823299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8823686Z outputs = self.model( 2025-11-03T16:34:44.8824027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8824387Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8824741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8825094Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8825421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8825758Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8826122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8826548Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8826714Z 2025-11-03T16:34:44.8826811Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8827148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8827451Z res = mod(**inputs) 2025-11-03T16:34:44.8827787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8828139Z outputs = self.model( 2025-11-03T16:34:44.8828484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8828842Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8829199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8829558Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8829882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8830221Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8830659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8831043Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8831173Z 2025-11-03T16:34:44.8831272Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8831621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8831944Z res = mod(**inputs) 2025-11-03T16:34:44.8832291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8832656Z outputs = self.model( 2025-11-03T16:34:44.8832997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8833370Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8833740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8834197Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8834554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8834912Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8835310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-11-03T16:34:44.8835714Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.8835847Z 2025-11-03T16:34:44.8835956Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8836293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8836607Z res = mod(**inputs) 2025-11-03T16:34:44.8836983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8837352Z outputs = self.model( 2025-11-03T16:34:44.8837703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8838066Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8838436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8838805Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8839139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8839502Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8839865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8840251Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8840635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8841001Z return func(*args, **kwargs) 2025-11-03T16:34:44.8841356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8841795Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8842001Z 2025-11-03T16:34:44.8842100Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8842443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8842760Z res = mod(**inputs) 2025-11-03T16:34:44.8843106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8843479Z outputs = self.model( 2025-11-03T16:34:44.8843869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8844243Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8844609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8844972Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8845331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8845681Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8846053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8846431Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8846811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8847175Z return func(*args, **kwargs) 2025-11-03T16:34:44.8847535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8847960Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8848106Z 2025-11-03T16:34:44.8848205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8848607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8848969Z res = mod(**inputs) 2025-11-03T16:34:44.8849317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8849687Z outputs = self.model( 2025-11-03T16:34:44.8850047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8855369Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8855919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8856305Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8856661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8857017Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8857407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8857798Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8858293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8858665Z return func(*args, **kwargs) 2025-11-03T16:34:44.8859042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8859429Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8859571Z 2025-11-03T16:34:44.8859654Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8859897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8860257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8860582Z res = mod(**inputs) 2025-11-03T16:34:44.8860944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8861316Z outputs = self.model( 2025-11-03T16:34:44.8861675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8862050Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8862464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8862829Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8863168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8863517Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8863918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8864318Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8864683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8865040Z return func(*args, **kwargs) 2025-11-03T16:34:44.8865390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8865775Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8866193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8866647Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8866830Z 2025-11-03T16:34:44.8866934Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8867290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8867605Z res = mod(**inputs) 2025-11-03T16:34:44.8867957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8868325Z outputs = self.model( 2025-11-03T16:34:44.8868679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8869079Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8869446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8869815Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8870149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8870497Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8870871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8871256Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8871649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8872012Z return func(*args, **kwargs) 2025-11-03T16:34:44.8872375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8872759Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8872891Z 2025-11-03T16:34:44.8873000Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8873354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8873657Z res = mod(**inputs) 2025-11-03T16:34:44.8874122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8874508Z outputs = self.model( 2025-11-03T16:34:44.8874909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8875332Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8875711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8876134Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8876493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8876896Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8877290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8877739Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8877923Z 2025-11-03T16:34:44.8878028Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8878393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8878724Z res = mod(**inputs) 2025-11-03T16:34:44.8879090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8879488Z outputs = self.model( 2025-11-03T16:34:44.8879864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8880267Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8880660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8881049Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8881411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8881784Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8882185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8882626Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8882835Z 2025-11-03T16:34:44.8882945Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8883319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8883663Z res = mod(**inputs) 2025-11-03T16:34:44.8884024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8884401Z outputs = self.model( 2025-11-03T16:34:44.8884768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8885161Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8885545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8885909Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8886246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8886604Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8886983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8887368Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8887502Z 2025-11-03T16:34:44.8887610Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8887955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8888273Z res = mod(**inputs) 2025-11-03T16:34:44.8888631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8889008Z outputs = self.model( 2025-11-03T16:34:44.8889360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8889746Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8890137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8890517Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8890870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8891227Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8891632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8892030Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8892427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8892798Z return func(*args, **kwargs) 2025-11-03T16:34:44.8893164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8893626Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8893834Z 2025-11-03T16:34:44.8893934Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8894288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8894598Z res = mod(**inputs) 2025-11-03T16:34:44.8894960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8895337Z outputs = self.model( 2025-11-03T16:34:44.8895684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8896049Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8896400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8896495Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8896705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8896787Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8897026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8897114Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8897349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8897446Z return func(*args, **kwargs) 2025-11-03T16:34:44.8897688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8897764Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8897767Z 2025-11-03T16:34:44.8897871Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8898055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8898114Z res = mod(**inputs) 2025-11-03T16:34:44.8898358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8898420Z outputs = self.model( 2025-11-03T16:34:44.8898664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8898732Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8898966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8899040Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8899266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8899350Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8899584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8899675Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8899918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8899986Z return func(*args, **kwargs) 2025-11-03T16:34:44.8900225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8900309Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8900312Z 2025-11-03T16:34:44.8900395Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8900494Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8900678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8900747Z res = mod(**inputs) 2025-11-03T16:34:44.8900981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8901053Z outputs = self.model( 2025-11-03T16:34:44.8901294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8901363Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8901609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8901679Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8901899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8901991Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8902239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8902324Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8902555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8902630Z return func(*args, **kwargs) 2025-11-03T16:34:44.8902872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8902992Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8903273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8903401Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8903406Z 2025-11-03T16:34:44.8903514Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8903703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8903770Z res = mod(**inputs) 2025-11-03T16:34:44.8904014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8904097Z outputs = self.model( 2025-11-03T16:34:44.8904338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8904406Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8904650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8904717Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8904957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8905034Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8905268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8905361Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8905699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8905776Z return func(*args, **kwargs) 2025-11-03T16:34:44.8906014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8906094Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8906104Z 2025-11-03T16:34:44.8906205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8906394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8906466Z res = mod(**inputs) 2025-11-03T16:34:44.8906709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8906785Z outputs = self.model( 2025-11-03T16:34:44.8907035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8907106Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8907347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8907415Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8907634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8907708Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8907967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8908093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8908097Z 2025-11-03T16:34:44.8908195Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8908392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8908453Z res = mod(**inputs) 2025-11-03T16:34:44.8908708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8908789Z outputs = self.model( 2025-11-03T16:34:44.8909030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8909106Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8909354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8909430Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8909645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8909720Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8909976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8910095Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8910099Z 2025-11-03T16:34:44.8910206Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8910396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8910463Z res = mod(**inputs) 2025-11-03T16:34:44.8910713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8910800Z outputs = self.model( 2025-11-03T16:34:44.8911064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8911135Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8911390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8911484Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8911703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8911790Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8912031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8912118Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8912122Z 2025-11-03T16:34:44.8912222Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8912414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8912482Z res = mod(**inputs) 2025-11-03T16:34:44.8912731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8912803Z outputs = self.model( 2025-11-03T16:34:44.8913054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8913132Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8913567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8913644Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8913954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8914085Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8914353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-11-03T16:34:44.8914433Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.8914437Z 2025-11-03T16:34:44.8914542Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8914755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8914820Z res = mod(**inputs) 2025-11-03T16:34:44.8915122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8915188Z outputs = self.model( 2025-11-03T16:34:44.8915432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8915513Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8915756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8915835Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8916050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8916135Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8916387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8916483Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8916730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8916803Z return func(*args, **kwargs) 2025-11-03T16:34:44.8917085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8917234Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8917238Z 2025-11-03T16:34:44.8917346Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8917538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8917629Z res = mod(**inputs) 2025-11-03T16:34:44.8917889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8917957Z outputs = self.model( 2025-11-03T16:34:44.8918213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8918285Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8918533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8918612Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8918832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8918918Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8919164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8919254Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8919505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8919576Z return func(*args, **kwargs) 2025-11-03T16:34:44.8919829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8919926Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8919932Z 2025-11-03T16:34:44.8920039Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8920234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8920296Z res = mod(**inputs) 2025-11-03T16:34:44.8920557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8920622Z outputs = self.model( 2025-11-03T16:34:44.8920876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8920962Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8921205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8921282Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8921498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8921578Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8921821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8921907Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8922149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8922219Z return func(*args, **kwargs) 2025-11-03T16:34:44.8922468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8922551Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8922554Z 2025-11-03T16:34:44.8922644Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8922761Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8922955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8923026Z res = mod(**inputs) 2025-11-03T16:34:44.8923273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8923347Z outputs = self.model( 2025-11-03T16:34:44.8923613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8923683Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8923936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8924003Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8924235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8924311Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8924540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8924628Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8924851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8924925Z return func(*args, **kwargs) 2025-11-03T16:34:44.8925158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8925261Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8925529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8925690Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8925695Z 2025-11-03T16:34:44.8925800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8925984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8926050Z res = mod(**inputs) 2025-11-03T16:34:44.8926290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8926354Z outputs = self.model( 2025-11-03T16:34:44.8926598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8926684Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8926922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8926988Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8927204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8927277Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8927507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8927599Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8927826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8927898Z return func(*args, **kwargs) 2025-11-03T16:34:44.8928129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8928205Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8928217Z 2025-11-03T16:34:44.8928314Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8928514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8928583Z res = mod(**inputs) 2025-11-03T16:34:44.8928819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8928891Z outputs = self.model( 2025-11-03T16:34:44.8929138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8929205Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8929443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8929510Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8929724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8929796Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8930032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8930152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8930155Z 2025-11-03T16:34:44.8930249Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8930439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8930497Z res = mod(**inputs) 2025-11-03T16:34:44.8930744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8930807Z outputs = self.model( 2025-11-03T16:34:44.8931050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8931139Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8931374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8931449Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8931655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8931728Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8931976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8932086Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8932105Z 2025-11-03T16:34:44.8932210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8932393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8932455Z res = mod(**inputs) 2025-11-03T16:34:44.8932701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8932765Z outputs = self.model( 2025-11-03T16:34:44.8933009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8933077Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8933320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8933387Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8933603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8933686Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8933918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8934002Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8934033Z 2025-11-03T16:34:44.8934129Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8934315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8934382Z res = mod(**inputs) 2025-11-03T16:34:44.8934618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8934707Z outputs = self.model( 2025-11-03T16:34:44.8934946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8935022Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8935255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8935326Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8935543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8935616Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8935856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8935940Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8936171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8936245Z return func(*args, **kwargs) 2025-11-03T16:34:44.8936479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8936628Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8936646Z 2025-11-03T16:34:44.8936742Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8936931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8936990Z res = mod(**inputs) 2025-11-03T16:34:44.8937226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8937296Z outputs = self.model( 2025-11-03T16:34:44.8937533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8937608Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8937856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8937923Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8938135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8938211Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8938448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8938532Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8938762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8938836Z return func(*args, **kwargs) 2025-11-03T16:34:44.8939069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8939153Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8939156Z 2025-11-03T16:34:44.8939250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8939437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8939498Z res = mod(**inputs) 2025-11-03T16:34:44.8939753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8939826Z outputs = self.model( 2025-11-03T16:34:44.8940061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8940135Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8940384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8940451Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8940674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8940746Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8940991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8941076Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8941303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8941375Z return func(*args, **kwargs) 2025-11-03T16:34:44.8941605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8941694Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8941697Z 2025-11-03T16:34:44.8941772Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8941876Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8942059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8942116Z res = mod(**inputs) 2025-11-03T16:34:44.8942382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8942445Z outputs = self.model( 2025-11-03T16:34:44.8942688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8942755Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8942991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8943066Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8943271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8943373Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8943604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8943696Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8943924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8943989Z return func(*args, **kwargs) 2025-11-03T16:34:44.8944232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8944323Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8944603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8944730Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8944733Z 2025-11-03T16:34:44.8944828Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8945022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8945085Z res = mod(**inputs) 2025-11-03T16:34:44.8945350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8945416Z outputs = self.model( 2025-11-03T16:34:44.8945663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8945730Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8945975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8946051Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8946259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8946338Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8946576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8946662Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8946898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8946965Z return func(*args, **kwargs) 2025-11-03T16:34:44.8947204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8947282Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8947285Z 2025-11-03T16:34:44.8947380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8947573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8947633Z res = mod(**inputs) 2025-11-03T16:34:44.8947877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8947961Z outputs = self.model( 2025-11-03T16:34:44.8948208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8948275Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8948512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8948587Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8948795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8948890Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8949119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8949231Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8949236Z 2025-11-03T16:34:44.8949341Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8949524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8949589Z res = mod(**inputs) 2025-11-03T16:34:44.8949826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8949897Z outputs = self.model( 2025-11-03T16:34:44.8950132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8950198Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8950438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8950504Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8950715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8950807Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8951039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8951157Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8951160Z 2025-11-03T16:34:44.8951255Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8951460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8951521Z res = mod(**inputs) 2025-11-03T16:34:44.8951760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8951830Z outputs = self.model( 2025-11-03T16:34:44.8952065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8952144Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8952379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8952454Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8952662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8952736Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8952983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8953060Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8953063Z 2025-11-03T16:34:44.8953166Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8953347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8953421Z res = mod(**inputs) 2025-11-03T16:34:44.8953671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8953735Z outputs = self.model( 2025-11-03T16:34:44.8954053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8954132Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8954382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8954450Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8954707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8954795Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8955050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-11-03T16:34:44.8955141Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.8955145Z 2025-11-03T16:34:44.8955251Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8955447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8955519Z res = mod(**inputs) 2025-11-03T16:34:44.8955809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8955882Z outputs = self.model( 2025-11-03T16:34:44.8956131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8956203Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8956472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8956542Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8956777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8956851Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8957100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8957204Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8957442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8957519Z return func(*args, **kwargs) 2025-11-03T16:34:44.8957760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8957910Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8957915Z 2025-11-03T16:34:44.8958013Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8958200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8958269Z res = mod(**inputs) 2025-11-03T16:34:44.8958509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8958580Z outputs = self.model( 2025-11-03T16:34:44.8958824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8958903Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8959143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8959212Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8959450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8959525Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8959769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8959854Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8960094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8960170Z return func(*args, **kwargs) 2025-11-03T16:34:44.8960410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8960514Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8960517Z 2025-11-03T16:34:44.8960615Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8960809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8960874Z res = mod(**inputs) 2025-11-03T16:34:44.8961120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8961194Z outputs = self.model( 2025-11-03T16:34:44.8961438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8961515Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8961760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8961829Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8962088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8962166Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8962456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8962543Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8962780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8962853Z return func(*args, **kwargs) 2025-11-03T16:34:44.8963113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8963203Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8963209Z 2025-11-03T16:34:44.8963285Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8963392Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8963578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8963641Z res = mod(**inputs) 2025-11-03T16:34:44.8963892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8963957Z outputs = self.model( 2025-11-03T16:34:44.8964212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8964281Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8964524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8964600Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8964813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8964896Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8965134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8965977Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8966220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8966290Z return func(*args, **kwargs) 2025-11-03T16:34:44.8966541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8966637Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8966929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8967087Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8967090Z 2025-11-03T16:34:44.8967189Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8967389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8967454Z res = mod(**inputs) 2025-11-03T16:34:44.8967705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8967771Z outputs = self.model( 2025-11-03T16:34:44.8968014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8968091Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8968331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8968407Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8968626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8968710Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8968968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8969054Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8969292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8969358Z return func(*args, **kwargs) 2025-11-03T16:34:44.8969620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8969699Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8969703Z 2025-11-03T16:34:44.8969800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8969996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8970056Z res = mod(**inputs) 2025-11-03T16:34:44.8970324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8970389Z outputs = self.model( 2025-11-03T16:34:44.8970640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8970710Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8970955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8971033Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8971251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8971336Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8971587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8971722Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8971726Z 2025-11-03T16:34:44.8971836Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8972031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8972100Z res = mod(**inputs) 2025-11-03T16:34:44.8972414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8972487Z outputs = self.model( 2025-11-03T16:34:44.8972729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8972816Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8973065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8973135Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8973357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8973434Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8973677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8973799Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8973802Z 2025-11-03T16:34:44.8973903Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8974100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8974163Z res = mod(**inputs) 2025-11-03T16:34:44.8974410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8974481Z outputs = self.model( 2025-11-03T16:34:44.8974743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8974821Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8975066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8975144Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8975357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8975451Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8975697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8975776Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8975779Z 2025-11-03T16:34:44.8975882Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8976073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8976136Z res = mod(**inputs) 2025-11-03T16:34:44.8976383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8976446Z outputs = self.model( 2025-11-03T16:34:44.8976698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8976770Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8977021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8977092Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8977308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8977393Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8977666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8977764Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8978002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8978071Z return func(*args, **kwargs) 2025-11-03T16:34:44.8978324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.8978470Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.8978492Z 2025-11-03T16:34:44.8978601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8978791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8978858Z res = mod(**inputs) 2025-11-03T16:34:44.8979112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8979179Z outputs = self.model( 2025-11-03T16:34:44.8979440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8979511Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8979766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8979835Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8980052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8980139Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8980385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8980484Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8980737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8980807Z return func(*args, **kwargs) 2025-11-03T16:34:44.8981055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.8981131Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.8981151Z 2025-11-03T16:34:44.8981260Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8981449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8981518Z res = mod(**inputs) 2025-11-03T16:34:44.8981763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8981830Z outputs = self.model( 2025-11-03T16:34:44.8982086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8982154Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8982404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8982473Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8982693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8982777Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8983029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8983125Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8983359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8983444Z return func(*args, **kwargs) 2025-11-03T16:34:44.8983695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.8983780Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.8983783Z 2025-11-03T16:34:44.8983870Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.8983971Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8984170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8984233Z res = mod(**inputs) 2025-11-03T16:34:44.8984503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8984578Z outputs = self.model( 2025-11-03T16:34:44.8984827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8984908Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8985159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8985229Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8985454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8985531Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8985783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8985872Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8986116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8986187Z return func(*args, **kwargs) 2025-11-03T16:34:44.8986451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.8986560Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.8986859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.8987000Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.8987023Z 2025-11-03T16:34:44.8987124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8987318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8987390Z res = mod(**inputs) 2025-11-03T16:34:44.8987641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8987718Z outputs = self.model( 2025-11-03T16:34:44.8987967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8988046Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8988292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8988362Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8988588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8988673Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8988933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 380, in forward 2025-11-03T16:34:44.8989020Z hidden_states, attn_weights = self.self_attn( 2025-11-03T16:34:44.8989256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.8989355Z return func(*args, **kwargs) 2025-11-03T16:34:44.8989599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.8989685Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.8989689Z 2025-11-03T16:34:44.8989787Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8989990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8990051Z res = mod(**inputs) 2025-11-03T16:34:44.8990300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8990394Z outputs = self.model( 2025-11-03T16:34:44.8990644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8990723Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8990977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8991048Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8991272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8991348Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8991609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8991727Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8991732Z 2025-11-03T16:34:44.8991840Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8992032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8992095Z res = mod(**inputs) 2025-11-03T16:34:44.8992393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8992460Z outputs = self.model( 2025-11-03T16:34:44.8992715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8992783Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8993039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8993119Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8993332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8993415Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8993655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-11-03T16:34:44.8993770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.8993782Z 2025-11-03T16:34:44.8993880Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8994140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8994215Z res = mod(**inputs) 2025-11-03T16:34:44.8994466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8994540Z outputs = self.model( 2025-11-03T16:34:44.8994788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8994859Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8995118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8995216Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8995438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8995513Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8995752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-11-03T16:34:44.8995837Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.8995842Z 2025-11-03T16:34:44.8995942Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8996137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8996217Z res = mod(**inputs) 2025-11-03T16:34:44.8996460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8996535Z outputs = self.model( 2025-11-03T16:34:44.8996776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1259, in forward 2025-11-03T16:34:44.8996851Z encoder_outputs = self.encoder( 2025-11-03T16:34:44.8997088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 882, in forward 2025-11-03T16:34:44.8997164Z layer_outputs = encoder_layer( 2025-11-03T16:34:44.8997378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.8997454Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.8997701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 395, in forward 2025-11-03T16:34:44.8997778Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.8997781Z 2025-11-03T16:34:44.8997886Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.8998200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.8998264Z res = mod(**inputs) 2025-11-03T16:34:44.8998520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.8998585Z outputs = self.model( 2025-11-03T16:34:44.8998863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.8998934Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.8999183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1099, in forward 2025-11-03T16:34:44.8999348Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-11-03T16:34:44.8999574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-11-03T16:34:44.8999654Z return func(*args, **kwargs) 2025-11-03T16:34:44.8999896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-11-03T16:34:44.9000111Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-11-03T16:34:44.9000423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-11-03T16:34:44.9000608Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-11-03T16:34:44.9000622Z 2025-11-03T16:34:44.9000723Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9000910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9000982Z res = mod(**inputs) 2025-11-03T16:34:44.9001244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9001318Z outputs = self.model( 2025-11-03T16:34:44.9001560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9001628Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9001879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1099, in forward 2025-11-03T16:34:44.9002037Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-11-03T16:34:44.9002282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-11-03T16:34:44.9002351Z return func(*args, **kwargs) 2025-11-03T16:34:44.9002593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-11-03T16:34:44.9002811Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-11-03T16:34:44.9003112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-11-03T16:34:44.9003298Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-11-03T16:34:44.9003302Z 2025-11-03T16:34:44.9003403Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9003602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9003667Z res = mod(**inputs) 2025-11-03T16:34:44.9003913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9003987Z outputs = self.model( 2025-11-03T16:34:44.9004251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9004331Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9004574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9004652Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9004881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9004958Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9005196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9005268Z return func(*args, **kwargs) 2025-11-03T16:34:44.9005514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9005616Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9005845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9005920Z return func(*args, **kwargs) 2025-11-03T16:34:44.9006158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9006311Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9006315Z 2025-11-03T16:34:44.9006417Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9006612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9006675Z res = mod(**inputs) 2025-11-03T16:34:44.9006918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9007009Z outputs = self.model( 2025-11-03T16:34:44.9007256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9007336Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9007581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9007651Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9007874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9007956Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9008208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9008276Z return func(*args, **kwargs) 2025-11-03T16:34:44.9008529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9008634Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9008863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9008936Z return func(*args, **kwargs) 2025-11-03T16:34:44.9009178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9009264Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9009268Z 2025-11-03T16:34:44.9009366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9009552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9009623Z res = mod(**inputs) 2025-11-03T16:34:44.9009869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9009960Z outputs = self.model( 2025-11-03T16:34:44.9010220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9010293Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9010557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9010643Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9010879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9010954Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9011192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9011257Z return func(*args, **kwargs) 2025-11-03T16:34:44.9011499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9011599Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9011830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9011904Z return func(*args, **kwargs) 2025-11-03T16:34:44.9012143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9012226Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9012229Z 2025-11-03T16:34:44.9012317Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9012416Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9012615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9012693Z res = mod(**inputs) 2025-11-03T16:34:44.9012935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9013007Z outputs = self.model( 2025-11-03T16:34:44.9013383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9013472Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9013729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9013808Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9014068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9014144Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9014383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9014451Z return func(*args, **kwargs) 2025-11-03T16:34:44.9014697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9014793Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9015024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9015098Z return func(*args, **kwargs) 2025-11-03T16:34:44.9015340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9015444Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9015727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9015864Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9015868Z 2025-11-03T16:34:44.9015994Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9016185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9016255Z res = mod(**inputs) 2025-11-03T16:34:44.9016504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9016576Z outputs = self.model( 2025-11-03T16:34:44.9016846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9016917Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9017171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9017240Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9017464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9017538Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9017774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9017847Z return func(*args, **kwargs) 2025-11-03T16:34:44.9018088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9018190Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9018417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9018493Z return func(*args, **kwargs) 2025-11-03T16:34:44.9018735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9018839Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9018844Z 2025-11-03T16:34:44.9018952Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9019139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9019208Z res = mod(**inputs) 2025-11-03T16:34:44.9019448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9019514Z outputs = self.model( 2025-11-03T16:34:44.9019766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9019853Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9020103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9020173Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9020394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9020468Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9020698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9020772Z return func(*args, **kwargs) 2025-11-03T16:34:44.9021013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9021124Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9021357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9021423Z return func(*args, **kwargs) 2025-11-03T16:34:44.9021672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9021842Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9021846Z 2025-11-03T16:34:44.9021953Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9022139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9022206Z res = mod(**inputs) 2025-11-03T16:34:44.9022464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9022530Z outputs = self.model( 2025-11-03T16:34:44.9022780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9022851Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9023098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9023166Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9023379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9023465Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9023696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9023768Z return func(*args, **kwargs) 2025-11-03T16:34:44.9024009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9024115Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9024352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9024417Z return func(*args, **kwargs) 2025-11-03T16:34:44.9024693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9024770Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9024773Z 2025-11-03T16:34:44.9024878Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9025066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9025126Z res = mod(**inputs) 2025-11-03T16:34:44.9025379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9025443Z outputs = self.model( 2025-11-03T16:34:44.9025707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9025774Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9026016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9026093Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9026303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9026384Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9026613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9026681Z return func(*args, **kwargs) 2025-11-03T16:34:44.9026930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9027036Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9027273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9027340Z return func(*args, **kwargs) 2025-11-03T16:34:44.9027606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9027688Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9027691Z 2025-11-03T16:34:44.9027769Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9027874Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9028076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9028147Z res = mod(**inputs) 2025-11-03T16:34:44.9028390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9028456Z outputs = self.model( 2025-11-03T16:34:44.9028707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9028780Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9029027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9029099Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9029324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9029397Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9029628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9029703Z return func(*args, **kwargs) 2025-11-03T16:34:44.9029942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9030053Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9030287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9030372Z return func(*args, **kwargs) 2025-11-03T16:34:44.9030619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9030713Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9030995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9031124Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9031128Z 2025-11-03T16:34:44.9031234Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9031438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9031500Z res = mod(**inputs) 2025-11-03T16:34:44.9031751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9031820Z outputs = self.model( 2025-11-03T16:34:44.9032072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9032140Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9032383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9032464Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9032685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9032768Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9032999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9033066Z return func(*args, **kwargs) 2025-11-03T16:34:44.9033333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9033436Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9033707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9033775Z return func(*args, **kwargs) 2025-11-03T16:34:44.9034116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9034212Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9034218Z 2025-11-03T16:34:44.9034325Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9034529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9034593Z res = mod(**inputs) 2025-11-03T16:34:44.9034870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9034938Z outputs = self.model( 2025-11-03T16:34:44.9035189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9035269Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9035587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9035680Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9035895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9035973Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9036220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9036307Z return func(*args, **kwargs) 2025-11-03T16:34:44.9036560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9036679Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9036683Z 2025-11-03T16:34:44.9036790Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9036980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9037043Z res = mod(**inputs) 2025-11-03T16:34:44.9037299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9037384Z outputs = self.model( 2025-11-03T16:34:44.9037642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9037715Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9037968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9038044Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9038262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9038345Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9038585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9038661Z return func(*args, **kwargs) 2025-11-03T16:34:44.9038909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9039028Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9039032Z 2025-11-03T16:34:44.9039139Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9039349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9039419Z res = mod(**inputs) 2025-11-03T16:34:44.9039669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9039735Z outputs = self.model( 2025-11-03T16:34:44.9040008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9040080Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9040335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9040408Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9040634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9040713Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9040952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9041029Z return func(*args, **kwargs) 2025-11-03T16:34:44.9041279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9041364Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9041368Z 2025-11-03T16:34:44.9041469Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9041663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9041736Z res = mod(**inputs) 2025-11-03T16:34:44.9041986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9042060Z outputs = self.model( 2025-11-03T16:34:44.9042335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9042408Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9042666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9042738Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9042963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9043039Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9043283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9043382Z return func(*args, **kwargs) 2025-11-03T16:34:44.9043635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9043743Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9043980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9044055Z return func(*args, **kwargs) 2025-11-03T16:34:44.9044301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9044453Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9044456Z 2025-11-03T16:34:44.9044565Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9044758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9044830Z res = mod(**inputs) 2025-11-03T16:34:44.9045086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9045158Z outputs = self.model( 2025-11-03T16:34:44.9045407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9045475Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9045718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9045785Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9046015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9046091Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9046318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9046397Z return func(*args, **kwargs) 2025-11-03T16:34:44.9046632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9046733Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9046961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9047026Z return func(*args, **kwargs) 2025-11-03T16:34:44.9047271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9047349Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9047352Z 2025-11-03T16:34:44.9047456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9047643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9047711Z res = mod(**inputs) 2025-11-03T16:34:44.9047949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9048030Z outputs = self.model( 2025-11-03T16:34:44.9048283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9048354Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9048602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9048671Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9048883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9048984Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9049216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9049288Z return func(*args, **kwargs) 2025-11-03T16:34:44.9049533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9049635Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9049868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9049933Z return func(*args, **kwargs) 2025-11-03T16:34:44.9050187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9050269Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9050273Z 2025-11-03T16:34:44.9050360Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9050458Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9050647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9050716Z res = mod(**inputs) 2025-11-03T16:34:44.9050981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9051054Z outputs = self.model( 2025-11-03T16:34:44.9051297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9051367Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9051628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9051699Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9051922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9051998Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9052237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9052304Z return func(*args, **kwargs) 2025-11-03T16:34:44.9052551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9052651Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9052881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9052952Z return func(*args, **kwargs) 2025-11-03T16:34:44.9053192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9053286Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9053569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9053697Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9053721Z 2025-11-03T16:34:44.9053828Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9054014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9054082Z res = mod(**inputs) 2025-11-03T16:34:44.9054329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9054393Z outputs = self.model( 2025-11-03T16:34:44.9054648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9054736Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9054988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9055058Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9055271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9055356Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9055586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9055658Z return func(*args, **kwargs) 2025-11-03T16:34:44.9055901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9055996Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9056232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9056298Z return func(*args, **kwargs) 2025-11-03T16:34:44.9056546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9056624Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9056628Z 2025-11-03T16:34:44.9056758Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9056946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9057007Z res = mod(**inputs) 2025-11-03T16:34:44.9057259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9057339Z outputs = self.model( 2025-11-03T16:34:44.9057590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9057660Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9057900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9057978Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9058195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9058277Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9058514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9058579Z return func(*args, **kwargs) 2025-11-03T16:34:44.9058825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 485, in forward 2025-11-03T16:34:44.9058903Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9058906Z 2025-11-03T16:34:44.9059012Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9059203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9059271Z res = mod(**inputs) 2025-11-03T16:34:44.9059513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9059599Z outputs = self.model( 2025-11-03T16:34:44.9059851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9059921Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9060171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9060241Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9060455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9060558Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9060787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9060864Z return func(*args, **kwargs) 2025-11-03T16:34:44.9061110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9061222Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9061451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9061517Z return func(*args, **kwargs) 2025-11-03T16:34:44.9061766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9061909Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9061914Z 2025-11-03T16:34:44.9062020Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9062209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9062271Z res = mod(**inputs) 2025-11-03T16:34:44.9062542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9062608Z outputs = self.model( 2025-11-03T16:34:44.9062857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9062926Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9063191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9063264Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9063476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9063560Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9063789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9063864Z return func(*args, **kwargs) 2025-11-03T16:34:44.9064105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9064211Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9064448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9064517Z return func(*args, **kwargs) 2025-11-03T16:34:44.9064764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9064843Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9064846Z 2025-11-03T16:34:44.9064952Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9065139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9065227Z res = mod(**inputs) 2025-11-03T16:34:44.9065479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9065543Z outputs = self.model( 2025-11-03T16:34:44.9065793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9065861Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9066105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9066183Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9066412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9066495Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9066726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9066798Z return func(*args, **kwargs) 2025-11-03T16:34:44.9067042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9067145Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9067381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9067447Z return func(*args, **kwargs) 2025-11-03T16:34:44.9067695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9067779Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9067782Z 2025-11-03T16:34:44.9067862Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9067969Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9068177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9068247Z res = mod(**inputs) 2025-11-03T16:34:44.9068489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9068553Z outputs = self.model( 2025-11-03T16:34:44.9068824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9068894Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9069141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9069211Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9069423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9069507Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9069740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9069817Z return func(*args, **kwargs) 2025-11-03T16:34:44.9070057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9070169Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9070408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9070474Z return func(*args, **kwargs) 2025-11-03T16:34:44.9070726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9070817Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9071118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9071248Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9071251Z 2025-11-03T16:34:44.9071348Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9071546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9071606Z res = mod(**inputs) 2025-11-03T16:34:44.9071857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9071921Z outputs = self.model( 2025-11-03T16:34:44.9072194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9072262Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9072507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9072587Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9072802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9072886Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9073117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9073185Z return func(*args, **kwargs) 2025-11-03T16:34:44.9073432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9073537Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9073775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9073842Z return func(*args, **kwargs) 2025-11-03T16:34:44.9074212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9074309Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9074313Z 2025-11-03T16:34:44.9074418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9074624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9074688Z res = mod(**inputs) 2025-11-03T16:34:44.9074971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9075044Z outputs = self.model( 2025-11-03T16:34:44.9075307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9075403Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9075657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9075739Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9075963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9076042Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9076297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9076368Z return func(*args, **kwargs) 2025-11-03T16:34:44.9076630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9076752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9076756Z 2025-11-03T16:34:44.9076868Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9077079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9077146Z res = mod(**inputs) 2025-11-03T16:34:44.9077405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9077470Z outputs = self.model( 2025-11-03T16:34:44.9077726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9077798Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9078051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9078147Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9078366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9078450Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9078692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9078760Z return func(*args, **kwargs) 2025-11-03T16:34:44.9079016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9079135Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9079138Z 2025-11-03T16:34:44.9079250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9079446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9079516Z res = mod(**inputs) 2025-11-03T16:34:44.9079768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9079834Z outputs = self.model( 2025-11-03T16:34:44.9080112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9080184Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9080442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9080511Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9080751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9080835Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9081069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9081144Z return func(*args, **kwargs) 2025-11-03T16:34:44.9081394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9081481Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9081485Z 2025-11-03T16:34:44.9081585Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9081776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9081845Z res = mod(**inputs) 2025-11-03T16:34:44.9082092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9082164Z outputs = self.model( 2025-11-03T16:34:44.9082412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9082484Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9082735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9082807Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9083059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9083136Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9083371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9083445Z return func(*args, **kwargs) 2025-11-03T16:34:44.9083692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9083799Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9084033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9084130Z return func(*args, **kwargs) 2025-11-03T16:34:44.9084374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9084526Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9084530Z 2025-11-03T16:34:44.9084639Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9084831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9084900Z res = mod(**inputs) 2025-11-03T16:34:44.9085148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9085215Z outputs = self.model( 2025-11-03T16:34:44.9085475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9085547Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9085801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9085871Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9086111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9086191Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9086429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9086507Z return func(*args, **kwargs) 2025-11-03T16:34:44.9086781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9086885Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9087124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9087192Z return func(*args, **kwargs) 2025-11-03T16:34:44.9087449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9087529Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9087533Z 2025-11-03T16:34:44.9087641Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9087835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9087897Z res = mod(**inputs) 2025-11-03T16:34:44.9088155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9088222Z outputs = self.model( 2025-11-03T16:34:44.9088479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9088550Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9088810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9088901Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9089119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9089204Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9089439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9089514Z return func(*args, **kwargs) 2025-11-03T16:34:44.9089759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9089970Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9090212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9090280Z return func(*args, **kwargs) 2025-11-03T16:34:44.9090534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9090619Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9090622Z 2025-11-03T16:34:44.9090711Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9090813Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9091006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9091080Z res = mod(**inputs) 2025-11-03T16:34:44.9091331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9091418Z outputs = self.model( 2025-11-03T16:34:44.9091655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9091724Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9091990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9092059Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9092275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9092349Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9092585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9092662Z return func(*args, **kwargs) 2025-11-03T16:34:44.9092898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9092998Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9093229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9093302Z return func(*args, **kwargs) 2025-11-03T16:34:44.9093541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9093632Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9093915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9094043Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9094046Z 2025-11-03T16:34:44.9094147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9094335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9094396Z res = mod(**inputs) 2025-11-03T16:34:44.9094649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9094726Z outputs = self.model( 2025-11-03T16:34:44.9094971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9095039Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9095281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9095350Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9095558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9095657Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9095884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9095954Z return func(*args, **kwargs) 2025-11-03T16:34:44.9096189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9096280Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9096510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9096574Z return func(*args, **kwargs) 2025-11-03T16:34:44.9096814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9096890Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9096893Z 2025-11-03T16:34:44.9096996Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9097178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9097237Z res = mod(**inputs) 2025-11-03T16:34:44.9097488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9097570Z outputs = self.model( 2025-11-03T16:34:44.9097815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9097883Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9098117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9098207Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9098415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9098497Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9098723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9098789Z return func(*args, **kwargs) 2025-11-03T16:34:44.9099031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9099133Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9099364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9099429Z return func(*args, **kwargs) 2025-11-03T16:34:44.9099672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9099816Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9099821Z 2025-11-03T16:34:44.9099918Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9100107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9100184Z res = mod(**inputs) 2025-11-03T16:34:44.9100436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9100499Z outputs = self.model( 2025-11-03T16:34:44.9100737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9100815Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9101052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9101126Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9101333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9101429Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9101660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9101727Z return func(*args, **kwargs) 2025-11-03T16:34:44.9101972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9102074Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9102306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9102374Z return func(*args, **kwargs) 2025-11-03T16:34:44.9102609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9102707Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9102710Z 2025-11-03T16:34:44.9102808Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9102998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9103060Z res = mod(**inputs) 2025-11-03T16:34:44.9103312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9103385Z outputs = self.model( 2025-11-03T16:34:44.9103625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9103700Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9103954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9104030Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9104239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9104311Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9104541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9104608Z return func(*args, **kwargs) 2025-11-03T16:34:44.9104844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9104945Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9105167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9105240Z return func(*args, **kwargs) 2025-11-03T16:34:44.9105472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9105559Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9105563Z 2025-11-03T16:34:44.9105636Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9105739Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9105943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9106004Z res = mod(**inputs) 2025-11-03T16:34:44.9106249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9106313Z outputs = self.model( 2025-11-03T16:34:44.9106557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9106628Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9106866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9106957Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9107165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9107249Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9107473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9107536Z return func(*args, **kwargs) 2025-11-03T16:34:44.9107779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9107880Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9108112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9108175Z return func(*args, **kwargs) 2025-11-03T16:34:44.9108415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9108505Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9108774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9108933Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9108936Z 2025-11-03T16:34:44.9109034Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9109228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9109290Z res = mod(**inputs) 2025-11-03T16:34:44.9109552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9109623Z outputs = self.model( 2025-11-03T16:34:44.9109868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9109945Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9110186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9110265Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9110475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9110548Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9110785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9110852Z return func(*args, **kwargs) 2025-11-03T16:34:44.9111095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9111200Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9111427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9111499Z return func(*args, **kwargs) 2025-11-03T16:34:44.9111756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9111843Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9111846Z 2025-11-03T16:34:44.9111944Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9112131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9112198Z res = mod(**inputs) 2025-11-03T16:34:44.9112441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9112533Z outputs = self.model( 2025-11-03T16:34:44.9112775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9112851Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9113096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9113164Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9113508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9113588Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9113830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9113899Z return func(*args, **kwargs) 2025-11-03T16:34:44.9114198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 502, in forward 2025-11-03T16:34:44.9114293Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9114297Z 2025-11-03T16:34:44.9114399Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9114600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9114707Z res = mod(**inputs) 2025-11-03T16:34:44.9114965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9115031Z outputs = self.model( 2025-11-03T16:34:44.9115275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9115351Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9115620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9115698Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9115917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9115990Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9116236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9116308Z return func(*args, **kwargs) 2025-11-03T16:34:44.9116581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9116698Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9116702Z 2025-11-03T16:34:44.9116800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9117005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9117068Z res = mod(**inputs) 2025-11-03T16:34:44.9117326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9117393Z outputs = self.model( 2025-11-03T16:34:44.9117647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9117742Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9117986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9118062Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9118273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9118355Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9118584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9118680Z return func(*args, **kwargs) 2025-11-03T16:34:44.9118924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9119041Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9119044Z 2025-11-03T16:34:44.9119150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9119338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9119407Z res = mod(**inputs) 2025-11-03T16:34:44.9119652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9119715Z outputs = self.model( 2025-11-03T16:34:44.9119963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9120035Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9120283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9120349Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9120577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9120667Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9120898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9120973Z return func(*args, **kwargs) 2025-11-03T16:34:44.9121229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9121306Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9121317Z 2025-11-03T16:34:44.9121418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9121613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9121682Z res = mod(**inputs) 2025-11-03T16:34:44.9121934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9122009Z outputs = self.model( 2025-11-03T16:34:44.9122256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9122324Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9122576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9122645Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9122862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9122938Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9123168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9123241Z return func(*args, **kwargs) 2025-11-03T16:34:44.9123499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9123598Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9123828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9123893Z return func(*args, **kwargs) 2025-11-03T16:34:44.9124138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9124278Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9124297Z 2025-11-03T16:34:44.9124404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9124589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9124658Z res = mod(**inputs) 2025-11-03T16:34:44.9124906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9124971Z outputs = self.model( 2025-11-03T16:34:44.9125223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9125290Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9125544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9125611Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9125828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9125909Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9126140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9126213Z return func(*args, **kwargs) 2025-11-03T16:34:44.9126471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9126576Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9126809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9126874Z return func(*args, **kwargs) 2025-11-03T16:34:44.9127144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9127220Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9127224Z 2025-11-03T16:34:44.9127324Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9127505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9127566Z res = mod(**inputs) 2025-11-03T16:34:44.9127817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9127882Z outputs = self.model( 2025-11-03T16:34:44.9128126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9128194Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9128437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9128505Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9128713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9128792Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9129017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9129107Z return func(*args, **kwargs) 2025-11-03T16:34:44.9129339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9129429Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9129659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9129725Z return func(*args, **kwargs) 2025-11-03T16:34:44.9129977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9130081Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9130085Z 2025-11-03T16:34:44.9130163Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9130265Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9130450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9130517Z res = mod(**inputs) 2025-11-03T16:34:44.9130755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9130824Z outputs = self.model( 2025-11-03T16:34:44.9131058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9131127Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9131368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9131435Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9131650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9131721Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9131964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9132037Z return func(*args, **kwargs) 2025-11-03T16:34:44.9132271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9132369Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9132608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9132672Z return func(*args, **kwargs) 2025-11-03T16:34:44.9132913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9133006Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9133280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9133408Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9133412Z 2025-11-03T16:34:44.9133514Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9133696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9133756Z res = mod(**inputs) 2025-11-03T16:34:44.9134005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9134068Z outputs = self.model( 2025-11-03T16:34:44.9134310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9134379Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9134612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9134706Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9134917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9134996Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9135225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9135297Z return func(*args, **kwargs) 2025-11-03T16:34:44.9135539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9135647Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9135880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9135944Z return func(*args, **kwargs) 2025-11-03T16:34:44.9136190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9136265Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9136269Z 2025-11-03T16:34:44.9136363Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9136553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9136611Z res = mod(**inputs) 2025-11-03T16:34:44.9136858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9136920Z outputs = self.model( 2025-11-03T16:34:44.9137165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9137232Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9137467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9137559Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9137772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9137853Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9138080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9138158Z return func(*args, **kwargs) 2025-11-03T16:34:44.9138399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9138502Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9138731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9138796Z return func(*args, **kwargs) 2025-11-03T16:34:44.9139031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9139175Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9139179Z 2025-11-03T16:34:44.9139274Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9139464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9139524Z res = mod(**inputs) 2025-11-03T16:34:44.9139768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9139831Z outputs = self.model( 2025-11-03T16:34:44.9140064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9140158Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9140396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9140469Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9140684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9140755Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9140989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9141053Z return func(*args, **kwargs) 2025-11-03T16:34:44.9141291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9141410Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9141644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9141710Z return func(*args, **kwargs) 2025-11-03T16:34:44.9141950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9142032Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9142035Z 2025-11-03T16:34:44.9142131Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9142328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9142389Z res = mod(**inputs) 2025-11-03T16:34:44.9142636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9142709Z outputs = self.model( 2025-11-03T16:34:44.9142951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9143027Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9143283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9143352Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9143565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9143636Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9143886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9143952Z return func(*args, **kwargs) 2025-11-03T16:34:44.9144195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9144297Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9144539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9144615Z return func(*args, **kwargs) 2025-11-03T16:34:44.9144857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9144945Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9144948Z 2025-11-03T16:34:44.9145024Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9145124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9145320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9145382Z res = mod(**inputs) 2025-11-03T16:34:44.9145632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9145695Z outputs = self.model( 2025-11-03T16:34:44.9145963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9146033Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9146277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9146352Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9146564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9146647Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9146878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9146960Z return func(*args, **kwargs) 2025-11-03T16:34:44.9147214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9147320Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9147568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9147632Z return func(*args, **kwargs) 2025-11-03T16:34:44.9147885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9147985Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9148273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9148410Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9148415Z 2025-11-03T16:34:44.9148523Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9148717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9148777Z res = mod(**inputs) 2025-11-03T16:34:44.9149047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9149121Z outputs = self.model( 2025-11-03T16:34:44.9149361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9149436Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9149689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9149758Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9149976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9150048Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9150280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9150346Z return func(*args, **kwargs) 2025-11-03T16:34:44.9150586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9150686Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9150907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9150980Z return func(*args, **kwargs) 2025-11-03T16:34:44.9151212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9151295Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9151298Z 2025-11-03T16:34:44.9151393Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9151575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9151670Z res = mod(**inputs) 2025-11-03T16:34:44.9151911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9151982Z outputs = self.model( 2025-11-03T16:34:44.9152225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9152291Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9152537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9152619Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9152832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9152904Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9153138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9153201Z return func(*args, **kwargs) 2025-11-03T16:34:44.9153432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9153550Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9153554Z 2025-11-03T16:34:44.9153650Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9153850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9153911Z res = mod(**inputs) 2025-11-03T16:34:44.9154232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9154309Z outputs = self.model( 2025-11-03T16:34:44.9154566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9154673Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9154925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9155004Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9155227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9155317Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9155552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9155619Z return func(*args, **kwargs) 2025-11-03T16:34:44.9155860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9155971Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9155976Z 2025-11-03T16:34:44.9156073Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9156268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9156329Z res = mod(**inputs) 2025-11-03T16:34:44.9156574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9156637Z outputs = self.model( 2025-11-03T16:34:44.9156878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9156956Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9157200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9157276Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9157502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9157583Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9157806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9157872Z return func(*args, **kwargs) 2025-11-03T16:34:44.9158111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9158197Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9158201Z 2025-11-03T16:34:44.9158303Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9158503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9158563Z res = mod(**inputs) 2025-11-03T16:34:44.9158806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9158871Z outputs = self.model( 2025-11-03T16:34:44.9159116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9159186Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9159428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9159498Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9159705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9159786Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9160016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9160088Z return func(*args, **kwargs) 2025-11-03T16:34:44.9160340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 511, in forward 2025-11-03T16:34:44.9160417Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9160421Z 2025-11-03T16:34:44.9160524Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9160707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9160775Z res = mod(**inputs) 2025-11-03T16:34:44.9161028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9161092Z outputs = self.model( 2025-11-03T16:34:44.9161338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9161407Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9161652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9161722Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9161936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9162008Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9162244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9162317Z return func(*args, **kwargs) 2025-11-03T16:34:44.9162551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9162654Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9162878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9162958Z return func(*args, **kwargs) 2025-11-03T16:34:44.9163200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9163338Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9163341Z 2025-11-03T16:34:44.9163444Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9163630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9163697Z res = mod(**inputs) 2025-11-03T16:34:44.9163934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9164016Z outputs = self.model( 2025-11-03T16:34:44.9164257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9164321Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9164564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9164630Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9164834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9164914Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9165138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9165208Z return func(*args, **kwargs) 2025-11-03T16:34:44.9165438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9165535Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9165763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9165827Z return func(*args, **kwargs) 2025-11-03T16:34:44.9166083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9166160Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9166164Z 2025-11-03T16:34:44.9166263Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9166461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9166521Z res = mod(**inputs) 2025-11-03T16:34:44.9166762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9166825Z outputs = self.model( 2025-11-03T16:34:44.9167067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9167135Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9167369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9167442Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9167646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9167724Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9167950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9168019Z return func(*args, **kwargs) 2025-11-03T16:34:44.9168254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9168347Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9168576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9168655Z return func(*args, **kwargs) 2025-11-03T16:34:44.9168896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9168977Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9168980Z 2025-11-03T16:34:44.9169055Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9169156Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9169339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9169404Z res = mod(**inputs) 2025-11-03T16:34:44.9169663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9169726Z outputs = self.model( 2025-11-03T16:34:44.9169974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9170044Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9170294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9170359Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9170575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9170651Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9170880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9170954Z return func(*args, **kwargs) 2025-11-03T16:34:44.9171190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9171292Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9171534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9171601Z return func(*args, **kwargs) 2025-11-03T16:34:44.9171844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9171937Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9172241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9172367Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9172372Z 2025-11-03T16:34:44.9172473Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9172656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9172716Z res = mod(**inputs) 2025-11-03T16:34:44.9172964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9173027Z outputs = self.model( 2025-11-03T16:34:44.9173274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9173341Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9173581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9173657Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9173866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9173948Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9174172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9174257Z return func(*args, **kwargs) 2025-11-03T16:34:44.9174494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9174587Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9174821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9174885Z return func(*args, **kwargs) 2025-11-03T16:34:44.9175131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9175232Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9175235Z 2025-11-03T16:34:44.9175332Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9175530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9175593Z res = mod(**inputs) 2025-11-03T16:34:44.9175848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9175915Z outputs = self.model( 2025-11-03T16:34:44.9176159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9176237Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9176482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9176559Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9176779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9176859Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9177094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9177181Z return func(*args, **kwargs) 2025-11-03T16:34:44.9177429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9177532Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9177770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9177851Z return func(*args, **kwargs) 2025-11-03T16:34:44.9178092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9178244Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9178248Z 2025-11-03T16:34:44.9178346Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9178540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9178601Z res = mod(**inputs) 2025-11-03T16:34:44.9178853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9178919Z outputs = self.model( 2025-11-03T16:34:44.9179163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9179243Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9179487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9179563Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9179778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9179959Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9180203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9180269Z return func(*args, **kwargs) 2025-11-03T16:34:44.9180517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9180620Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9180852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9180925Z return func(*args, **kwargs) 2025-11-03T16:34:44.9181183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9181269Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9181274Z 2025-11-03T16:34:44.9181373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9181571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9181631Z res = mod(**inputs) 2025-11-03T16:34:44.9181875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9181949Z outputs = self.model( 2025-11-03T16:34:44.9182195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9182275Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9182521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9182592Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9182822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9182904Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9183174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9183245Z return func(*args, **kwargs) 2025-11-03T16:34:44.9183496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9183602Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9183857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9183941Z return func(*args, **kwargs) 2025-11-03T16:34:44.9184192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9184297Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9184301Z 2025-11-03T16:34:44.9184381Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9184480Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9184680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9184741Z res = mod(**inputs) 2025-11-03T16:34:44.9184998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9185064Z outputs = self.model( 2025-11-03T16:34:44.9185311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9185393Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9185641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9185718Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9185953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9186039Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9186268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9186335Z return func(*args, **kwargs) 2025-11-03T16:34:44.9186583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9186686Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9186925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9187008Z return func(*args, **kwargs) 2025-11-03T16:34:44.9187252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9187355Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9187635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9187770Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9187773Z 2025-11-03T16:34:44.9187871Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9188066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9188127Z res = mod(**inputs) 2025-11-03T16:34:44.9188370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9188445Z outputs = self.model( 2025-11-03T16:34:44.9188687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9188764Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9189024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9189093Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9189312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9189386Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9189647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9189715Z return func(*args, **kwargs) 2025-11-03T16:34:44.9189962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9190069Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9190298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9190376Z return func(*args, **kwargs) 2025-11-03T16:34:44.9190615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9190698Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9190701Z 2025-11-03T16:34:44.9190798Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9190990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9191059Z res = mod(**inputs) 2025-11-03T16:34:44.9191302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9191377Z outputs = self.model( 2025-11-03T16:34:44.9191616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9191700Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9191949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9192018Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9192236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9192310Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9192546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9192628Z return func(*args, **kwargs) 2025-11-03T16:34:44.9192866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9192998Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9193003Z 2025-11-03T16:34:44.9193099Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9193290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9193348Z res = mod(**inputs) 2025-11-03T16:34:44.9193585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9193655Z outputs = self.model( 2025-11-03T16:34:44.9193893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9193968Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9194283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9194357Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9194582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9194692Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9194947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9195017Z return func(*args, **kwargs) 2025-11-03T16:34:44.9195272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9195416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9195420Z 2025-11-03T16:34:44.9195519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9195710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9195770Z res = mod(**inputs) 2025-11-03T16:34:44.9196014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9196081Z outputs = self.model( 2025-11-03T16:34:44.9196321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9196397Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9196633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9196706Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9196914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9196996Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9197218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9197283Z return func(*args, **kwargs) 2025-11-03T16:34:44.9197546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9197621Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9197624Z 2025-11-03T16:34:44.9197727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9197907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9197967Z res = mod(**inputs) 2025-11-03T16:34:44.9198213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9198277Z outputs = self.model( 2025-11-03T16:34:44.9198534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9198603Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9198845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9198921Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9199127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9199209Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9199433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9199507Z return func(*args, **kwargs) 2025-11-03T16:34:44.9199745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9199840Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9200071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9200135Z return func(*args, **kwargs) 2025-11-03T16:34:44.9200394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9200534Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9200537Z 2025-11-03T16:34:44.9200631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9200822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9200896Z res = mod(**inputs) 2025-11-03T16:34:44.9201144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9201209Z outputs = self.model( 2025-11-03T16:34:44.9201460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9201528Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9201774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9201849Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9202062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9202145Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9202379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9202444Z return func(*args, **kwargs) 2025-11-03T16:34:44.9202689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9202784Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9203020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9203104Z return func(*args, **kwargs) 2025-11-03T16:34:44.9203342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9203427Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9203430Z 2025-11-03T16:34:44.9203528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9203720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9203783Z res = mod(**inputs) 2025-11-03T16:34:44.9204030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9204112Z outputs = self.model( 2025-11-03T16:34:44.9204353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9204432Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9204678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9204752Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9204964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9205036Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9205277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9205342Z return func(*args, **kwargs) 2025-11-03T16:34:44.9205589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9205683Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9205922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9206003Z return func(*args, **kwargs) 2025-11-03T16:34:44.9206245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9206335Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9206339Z 2025-11-03T16:34:44.9206415Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9206520Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9206721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9206782Z res = mod(**inputs) 2025-11-03T16:34:44.9207037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9207102Z outputs = self.model( 2025-11-03T16:34:44.9207359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9207433Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9207681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9207756Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9207973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9208063Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9208292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9208365Z return func(*args, **kwargs) 2025-11-03T16:34:44.9208606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9208700Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9208948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9209012Z return func(*args, **kwargs) 2025-11-03T16:34:44.9209252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9209342Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9209610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9209741Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9209762Z 2025-11-03T16:34:44.9209857Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9210046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9210109Z res = mod(**inputs) 2025-11-03T16:34:44.9210356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9210423Z outputs = self.model( 2025-11-03T16:34:44.9210663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9210737Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9210974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9211051Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9211260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9211334Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9211570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9211634Z return func(*args, **kwargs) 2025-11-03T16:34:44.9211895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9211986Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9212210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9212282Z return func(*args, **kwargs) 2025-11-03T16:34:44.9212541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9212628Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9212632Z 2025-11-03T16:34:44.9212728Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9212921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9212983Z res = mod(**inputs) 2025-11-03T16:34:44.9213333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9213416Z outputs = self.model( 2025-11-03T16:34:44.9213665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9213758Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9214038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9214110Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9214337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9214414Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9214659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9214773Z return func(*args, **kwargs) 2025-11-03T16:34:44.9215028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 485, in forward 2025-11-03T16:34:44.9215104Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9215108Z 2025-11-03T16:34:44.9215209Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9215409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9215469Z res = mod(**inputs) 2025-11-03T16:34:44.9215726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9215825Z outputs = self.model( 2025-11-03T16:34:44.9216069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9216147Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9216390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9216468Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9216680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9216752Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9216993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9217059Z return func(*args, **kwargs) 2025-11-03T16:34:44.9217306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9217410Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9217673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9217740Z return func(*args, **kwargs) 2025-11-03T16:34:44.9217980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9218131Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9218134Z 2025-11-03T16:34:44.9218257Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9218453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9218517Z res = mod(**inputs) 2025-11-03T16:34:44.9218764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9218834Z outputs = self.model( 2025-11-03T16:34:44.9219079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9219158Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9219401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9219475Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9219686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9219760Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9219995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9220061Z return func(*args, **kwargs) 2025-11-03T16:34:44.9220309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9220428Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9220660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9220732Z return func(*args, **kwargs) 2025-11-03T16:34:44.9220973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9221054Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9221057Z 2025-11-03T16:34:44.9221156Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9221351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9221430Z res = mod(**inputs) 2025-11-03T16:34:44.9221676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9221746Z outputs = self.model( 2025-11-03T16:34:44.9221991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9222068Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9222307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9222375Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9222599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9222674Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9222914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9222981Z return func(*args, **kwargs) 2025-11-03T16:34:44.9223220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9223329Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9223577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9223650Z return func(*args, **kwargs) 2025-11-03T16:34:44.9223889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9223976Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9223995Z 2025-11-03T16:34:44.9224073Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9224173Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9224369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9224429Z res = mod(**inputs) 2025-11-03T16:34:44.9224684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9224749Z outputs = self.model( 2025-11-03T16:34:44.9224997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9225075Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9225319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9225395Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9225612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9225688Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9225924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9225992Z return func(*args, **kwargs) 2025-11-03T16:34:44.9226259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9226360Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9226597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9226662Z return func(*args, **kwargs) 2025-11-03T16:34:44.9226902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9227003Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9227278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9227429Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9227433Z 2025-11-03T16:34:44.9227532Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9227722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9227791Z res = mod(**inputs) 2025-11-03T16:34:44.9228038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9228110Z outputs = self.model( 2025-11-03T16:34:44.9228354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9228432Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9228675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9228745Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9228972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9229054Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9229308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9229374Z return func(*args, **kwargs) 2025-11-03T16:34:44.9229606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9229712Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9229949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9230021Z return func(*args, **kwargs) 2025-11-03T16:34:44.9230257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9230332Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9230344Z 2025-11-03T16:34:44.9230441Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9230628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9230695Z res = mod(**inputs) 2025-11-03T16:34:44.9230935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9231006Z outputs = self.model( 2025-11-03T16:34:44.9231244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9231311Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9231554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9231622Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9231843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9231933Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9232158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9232230Z return func(*args, **kwargs) 2025-11-03T16:34:44.9232465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9232587Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9232590Z 2025-11-03T16:34:44.9232684Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9232890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9232949Z res = mod(**inputs) 2025-11-03T16:34:44.9233185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9233256Z outputs = self.model( 2025-11-03T16:34:44.9233490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9233566Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9233810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9233884Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9234187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9234276Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9234532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9234600Z return func(*args, **kwargs) 2025-11-03T16:34:44.9257314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9257683Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9257690Z 2025-11-03T16:34:44.9257813Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9258034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9258104Z res = mod(**inputs) 2025-11-03T16:34:44.9258464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9258540Z outputs = self.model( 2025-11-03T16:34:44.9258792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9258881Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9259131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9259214Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9259433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9259524Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9259764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9259836Z return func(*args, **kwargs) 2025-11-03T16:34:44.9260087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9260171Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9260175Z 2025-11-03T16:34:44.9260285Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9260476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9260571Z res = mod(**inputs) 2025-11-03T16:34:44.9260821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9260889Z outputs = self.model( 2025-11-03T16:34:44.9261132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9261205Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9261448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9261517Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9261753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9261841Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9262068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9262147Z return func(*args, **kwargs) 2025-11-03T16:34:44.9262385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9262484Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9262719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9262787Z return func(*args, **kwargs) 2025-11-03T16:34:44.9263033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9263180Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9263184Z 2025-11-03T16:34:44.9263292Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9263484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9263594Z res = mod(**inputs) 2025-11-03T16:34:44.9263844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9263910Z outputs = self.model( 2025-11-03T16:34:44.9264154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9264243Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9264485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9264562Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9264769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9264851Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9265078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9265143Z return func(*args, **kwargs) 2025-11-03T16:34:44.9265386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9265481Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9265714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9265780Z return func(*args, **kwargs) 2025-11-03T16:34:44.9266023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9266103Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9266107Z 2025-11-03T16:34:44.9266205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9266418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9266481Z res = mod(**inputs) 2025-11-03T16:34:44.9266726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9266789Z outputs = self.model( 2025-11-03T16:34:44.9267026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9267107Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9267342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9267438Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9267644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9267721Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9267954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9268018Z return func(*args, **kwargs) 2025-11-03T16:34:44.9268260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9268352Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9268583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9268647Z return func(*args, **kwargs) 2025-11-03T16:34:44.9268887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9268975Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9268978Z 2025-11-03T16:34:44.9269059Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9269162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9269362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9269425Z res = mod(**inputs) 2025-11-03T16:34:44.9269674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9269740Z outputs = self.model( 2025-11-03T16:34:44.9270000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9270070Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9270309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9270385Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9270594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9270679Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9270913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9270989Z return func(*args, **kwargs) 2025-11-03T16:34:44.9271225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9271319Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9271555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9271623Z return func(*args, **kwargs) 2025-11-03T16:34:44.9271869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9271981Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9272257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9272396Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9272399Z 2025-11-03T16:34:44.9272496Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9272693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9272754Z res = mod(**inputs) 2025-11-03T16:34:44.9273002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9273081Z outputs = self.model( 2025-11-03T16:34:44.9273319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9273398Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9273645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9273722Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9273945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9274105Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9274363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9274433Z return func(*args, **kwargs) 2025-11-03T16:34:44.9274689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9274786Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9275032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9275128Z return func(*args, **kwargs) 2025-11-03T16:34:44.9275385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9275480Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9275484Z 2025-11-03T16:34:44.9275589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9275817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9275885Z res = mod(**inputs) 2025-11-03T16:34:44.9276144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9276224Z outputs = self.model( 2025-11-03T16:34:44.9276490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9276574Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9276826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9276899Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9277128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9277206Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9277457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9277525Z return func(*args, **kwargs) 2025-11-03T16:34:44.9277780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9277891Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9278148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9278226Z return func(*args, **kwargs) 2025-11-03T16:34:44.9278469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9278626Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9278630Z 2025-11-03T16:34:44.9278730Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9278925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9278997Z res = mod(**inputs) 2025-11-03T16:34:44.9279264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9279340Z outputs = self.model( 2025-11-03T16:34:44.9279590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9279672Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9279919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9279990Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9280216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9280293Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9280541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9280615Z return func(*args, **kwargs) 2025-11-03T16:34:44.9280866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9280983Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9281250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9281330Z return func(*args, **kwargs) 2025-11-03T16:34:44.9281587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9281672Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9281684Z 2025-11-03T16:34:44.9281806Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9282007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9282084Z res = mod(**inputs) 2025-11-03T16:34:44.9282343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9282420Z outputs = self.model( 2025-11-03T16:34:44.9282680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9282754Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9283016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9283091Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9283324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9283405Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9283647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9283764Z return func(*args, **kwargs) 2025-11-03T16:34:44.9284046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9284200Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9284479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9284548Z return func(*args, **kwargs) 2025-11-03T16:34:44.9284807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9284903Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9284907Z 2025-11-03T16:34:44.9284991Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9285102Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9285318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9285383Z res = mod(**inputs) 2025-11-03T16:34:44.9285644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9285713Z outputs = self.model( 2025-11-03T16:34:44.9285976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9286048Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9286301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9286382Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9286606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9286695Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9286933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9287007Z return func(*args, **kwargs) 2025-11-03T16:34:44.9287312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9287440Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9287688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9287757Z return func(*args, **kwargs) 2025-11-03T16:34:44.9288028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9288129Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9288421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9288582Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9288586Z 2025-11-03T16:34:44.9288686Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9288894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9288961Z res = mod(**inputs) 2025-11-03T16:34:44.9289221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9289289Z outputs = self.model( 2025-11-03T16:34:44.9289543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9289627Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9289876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9289957Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9290176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9290275Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9290524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9290594Z return func(*args, **kwargs) 2025-11-03T16:34:44.9290855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9290962Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9291212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9291289Z return func(*args, **kwargs) 2025-11-03T16:34:44.9291637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9291746Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9291750Z 2025-11-03T16:34:44.9291853Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9292059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9292122Z res = mod(**inputs) 2025-11-03T16:34:44.9292369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9292444Z outputs = self.model( 2025-11-03T16:34:44.9292686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9292764Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9293007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9293080Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9293299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9293376Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9293741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9293808Z return func(*args, **kwargs) 2025-11-03T16:34:44.9294061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 502, in forward 2025-11-03T16:34:44.9294140Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9294167Z 2025-11-03T16:34:44.9294266Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9294462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9294526Z res = mod(**inputs) 2025-11-03T16:34:44.9294777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9294846Z outputs = self.model( 2025-11-03T16:34:44.9295089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9295167Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9295412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9295491Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9295705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9295790Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9296023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9296090Z return func(*args, **kwargs) 2025-11-03T16:34:44.9296339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9296485Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9296488Z 2025-11-03T16:34:44.9296594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9296787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9296850Z res = mod(**inputs) 2025-11-03T16:34:44.9297099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9297165Z outputs = self.model( 2025-11-03T16:34:44.9297414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9297505Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9297749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9297830Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9298044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9298127Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9298358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9298431Z return func(*args, **kwargs) 2025-11-03T16:34:44.9298671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9298788Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9298792Z 2025-11-03T16:34:44.9298900Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9299090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9299160Z res = mod(**inputs) 2025-11-03T16:34:44.9299417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9299484Z outputs = self.model( 2025-11-03T16:34:44.9299732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9299801Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9300064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9300136Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9300357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9300430Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9300660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9300735Z return func(*args, **kwargs) 2025-11-03T16:34:44.9300973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9301058Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9301061Z 2025-11-03T16:34:44.9301157Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9301344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9301413Z res = mod(**inputs) 2025-11-03T16:34:44.9301653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9301727Z outputs = self.model( 2025-11-03T16:34:44.9301969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9302056Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9302310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9302378Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9302601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9302677Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9302920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9302986Z return func(*args, **kwargs) 2025-11-03T16:34:44.9303250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9303354Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9303586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9303664Z return func(*args, **kwargs) 2025-11-03T16:34:44.9303903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9304048Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9304060Z 2025-11-03T16:34:44.9304158Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9304348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9304419Z res = mod(**inputs) 2025-11-03T16:34:44.9304663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9304738Z outputs = self.model( 2025-11-03T16:34:44.9304982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9305068Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9305327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9305396Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9305607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9305707Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9305941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9306015Z return func(*args, **kwargs) 2025-11-03T16:34:44.9306255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9306360Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9306598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9306662Z return func(*args, **kwargs) 2025-11-03T16:34:44.9306909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9306984Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9306987Z 2025-11-03T16:34:44.9307089Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9307276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9307337Z res = mod(**inputs) 2025-11-03T16:34:44.9307586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9307650Z outputs = self.model( 2025-11-03T16:34:44.9307912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9307979Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9308215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9308288Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9308494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9308575Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9308799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9308878Z return func(*args, **kwargs) 2025-11-03T16:34:44.9309117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9309208Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9309443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9309505Z return func(*args, **kwargs) 2025-11-03T16:34:44.9309744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9309823Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9309826Z 2025-11-03T16:34:44.9309903Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9310008Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9310192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9310258Z res = mod(**inputs) 2025-11-03T16:34:44.9310495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9310559Z outputs = self.model( 2025-11-03T16:34:44.9310817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9310885Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9311125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9311191Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9311418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9311496Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9311725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9311796Z return func(*args, **kwargs) 2025-11-03T16:34:44.9312036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9312141Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9312369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9312432Z return func(*args, **kwargs) 2025-11-03T16:34:44.9312678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9312775Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9313059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9313189Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9313192Z 2025-11-03T16:34:44.9313478Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9313723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9313784Z res = mod(**inputs) 2025-11-03T16:34:44.9314090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9314165Z outputs = self.model( 2025-11-03T16:34:44.9314420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9314494Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9314742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9314856Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9315080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9315170Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9315412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9315481Z return func(*args, **kwargs) 2025-11-03T16:34:44.9315735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9315831Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9316068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9316138Z return func(*args, **kwargs) 2025-11-03T16:34:44.9316386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9316467Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9316470Z 2025-11-03T16:34:44.9316567Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9316786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9316847Z res = mod(**inputs) 2025-11-03T16:34:44.9317092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9317155Z outputs = self.model( 2025-11-03T16:34:44.9317420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9317497Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9317735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9317811Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9318021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9318094Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9318332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9318398Z return func(*args, **kwargs) 2025-11-03T16:34:44.9318641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9318741Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9318973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9319039Z return func(*args, **kwargs) 2025-11-03T16:34:44.9319278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9319424Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9319452Z 2025-11-03T16:34:44.9319547Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9319738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9319797Z res = mod(**inputs) 2025-11-03T16:34:44.9320034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9320103Z outputs = self.model( 2025-11-03T16:34:44.9320340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9320417Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9320670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9320745Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9320954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9321030Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9321266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9321331Z return func(*args, **kwargs) 2025-11-03T16:34:44.9321576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9321681Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9321909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9321985Z return func(*args, **kwargs) 2025-11-03T16:34:44.9322224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9322309Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9322312Z 2025-11-03T16:34:44.9322427Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9322621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9322684Z res = mod(**inputs) 2025-11-03T16:34:44.9322926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9323000Z outputs = self.model( 2025-11-03T16:34:44.9323261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9323338Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9323577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9323645Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9323865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9323939Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9324172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9324238Z return func(*args, **kwargs) 2025-11-03T16:34:44.9324478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9324588Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9324815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9324890Z return func(*args, **kwargs) 2025-11-03T16:34:44.9325127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9325243Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9325249Z 2025-11-03T16:34:44.9325327Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9325424Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9325618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9325679Z res = mod(**inputs) 2025-11-03T16:34:44.9325932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9325997Z outputs = self.model( 2025-11-03T16:34:44.9326238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9326333Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9326574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9326660Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9326866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9326942Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9327173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9327236Z return func(*args, **kwargs) 2025-11-03T16:34:44.9327475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9327575Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9327806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9327867Z return func(*args, **kwargs) 2025-11-03T16:34:44.9328118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9328219Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9328486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9328612Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9328616Z 2025-11-03T16:34:44.9328722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9328906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9328976Z res = mod(**inputs) 2025-11-03T16:34:44.9329227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9329300Z outputs = self.model( 2025-11-03T16:34:44.9329544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9329623Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9329862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9329928Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9330143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9330218Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9330449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9330515Z return func(*args, **kwargs) 2025-11-03T16:34:44.9330747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9330868Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9331093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9331163Z return func(*args, **kwargs) 2025-11-03T16:34:44.9331396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9331478Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9331481Z 2025-11-03T16:34:44.9331577Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9331760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9331844Z res = mod(**inputs) 2025-11-03T16:34:44.9332079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9332147Z outputs = self.model( 2025-11-03T16:34:44.9332382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9332447Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9332691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9332757Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9332976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9333049Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9333272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9333344Z return func(*args, **kwargs) 2025-11-03T16:34:44.9333574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9333710Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9333714Z 2025-11-03T16:34:44.9333810Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9334000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9334058Z res = mod(**inputs) 2025-11-03T16:34:44.9334305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9334378Z outputs = self.model( 2025-11-03T16:34:44.9334610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9334683Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9334919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9334987Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9335203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9335278Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9335505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9335567Z return func(*args, **kwargs) 2025-11-03T16:34:44.9335801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9335916Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9335921Z 2025-11-03T16:34:44.9336016Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9336203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9336278Z res = mod(**inputs) 2025-11-03T16:34:44.9336518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9336581Z outputs = self.model( 2025-11-03T16:34:44.9336823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9336897Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9337132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9337203Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9337408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9337498Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9337725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9337792Z return func(*args, **kwargs) 2025-11-03T16:34:44.9338033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9338110Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9338114Z 2025-11-03T16:34:44.9338212Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9338393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9338452Z res = mod(**inputs) 2025-11-03T16:34:44.9338693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9338756Z outputs = self.model( 2025-11-03T16:34:44.9338994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9339062Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9339312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9339387Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9339595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9339678Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9339917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9339982Z return func(*args, **kwargs) 2025-11-03T16:34:44.9340223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 511, in forward 2025-11-03T16:34:44.9340306Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9340310Z 2025-11-03T16:34:44.9340404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9340600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9340659Z res = mod(**inputs) 2025-11-03T16:34:44.9340897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9340968Z outputs = self.model( 2025-11-03T16:34:44.9341207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9341279Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9341514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9341583Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9341801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9341890Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9342120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9342184Z return func(*args, **kwargs) 2025-11-03T16:34:44.9342421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9342513Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9342734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9342805Z return func(*args, **kwargs) 2025-11-03T16:34:44.9343057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9343204Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9343209Z 2025-11-03T16:34:44.9343303Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9343487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9343555Z res = mod(**inputs) 2025-11-03T16:34:44.9343791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9343862Z outputs = self.model( 2025-11-03T16:34:44.9344094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9344167Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9344402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9344468Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9344683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9344777Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9345009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9345075Z return func(*args, **kwargs) 2025-11-03T16:34:44.9345304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9345429Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9345653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9345727Z return func(*args, **kwargs) 2025-11-03T16:34:44.9345956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9346032Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9346041Z 2025-11-03T16:34:44.9346137Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9346319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9346385Z res = mod(**inputs) 2025-11-03T16:34:44.9346619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9346688Z outputs = self.model( 2025-11-03T16:34:44.9346923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9347006Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9347239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9347305Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9347540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9347612Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9347846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9347910Z return func(*args, **kwargs) 2025-11-03T16:34:44.9348142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9348243Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9348469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9348559Z return func(*args, **kwargs) 2025-11-03T16:34:44.9348796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9348885Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9348888Z 2025-11-03T16:34:44.9348963Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9349058Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9349253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9349311Z res = mod(**inputs) 2025-11-03T16:34:44.9349566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9349630Z outputs = self.model( 2025-11-03T16:34:44.9349870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9349946Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9350188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9350262Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9350486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9350559Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9350790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9350853Z return func(*args, **kwargs) 2025-11-03T16:34:44.9351107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9351199Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9351429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9351494Z return func(*args, **kwargs) 2025-11-03T16:34:44.9351730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9351830Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9352100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9352230Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9352233Z 2025-11-03T16:34:44.9352329Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9352517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9352577Z res = mod(**inputs) 2025-11-03T16:34:44.9352815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9352886Z outputs = self.model( 2025-11-03T16:34:44.9353122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9353215Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9353453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9353519Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9353735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9353810Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9354143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9354242Z return func(*args, **kwargs) 2025-11-03T16:34:44.9354495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9354602Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9354849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9354926Z return func(*args, **kwargs) 2025-11-03T16:34:44.9355184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9355280Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9355283Z 2025-11-03T16:34:44.9355380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9355568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9355639Z res = mod(**inputs) 2025-11-03T16:34:44.9355894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9355970Z outputs = self.model( 2025-11-03T16:34:44.9356257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9356330Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9356593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9356663Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9356905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9356984Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9357226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9357306Z return func(*args, **kwargs) 2025-11-03T16:34:44.9357552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9357672Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9357915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9357992Z return func(*args, **kwargs) 2025-11-03T16:34:44.9358239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9358390Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9358394Z 2025-11-03T16:34:44.9358504Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9358703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9358776Z res = mod(**inputs) 2025-11-03T16:34:44.9359030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9359114Z outputs = self.model( 2025-11-03T16:34:44.9359375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9359445Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9359700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9359769Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9359993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9360070Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9360324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9360399Z return func(*args, **kwargs) 2025-11-03T16:34:44.9360643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9360760Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9361002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9361069Z return func(*args, **kwargs) 2025-11-03T16:34:44.9361323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9361402Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9361405Z 2025-11-03T16:34:44.9361512Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9361704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9361772Z res = mod(**inputs) 2025-11-03T16:34:44.9362021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9362103Z outputs = self.model( 2025-11-03T16:34:44.9362359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9362429Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9362683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9362770Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9362989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9363074Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9363313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9363387Z return func(*args, **kwargs) 2025-11-03T16:34:44.9363636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9363740Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9363983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9364050Z return func(*args, **kwargs) 2025-11-03T16:34:44.9364302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9364386Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9364390Z 2025-11-03T16:34:44.9364478Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9364578Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9364775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9364860Z res = mod(**inputs) 2025-11-03T16:34:44.9365100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9365171Z outputs = self.model( 2025-11-03T16:34:44.9365407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9365473Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9365717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9365782Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9365994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9366084Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9366306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9366378Z return func(*args, **kwargs) 2025-11-03T16:34:44.9366609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9366718Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9366939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9367011Z return func(*args, **kwargs) 2025-11-03T16:34:44.9367243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9367337Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9367613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9367738Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9367743Z 2025-11-03T16:34:44.9367868Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9368053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9368112Z res = mod(**inputs) 2025-11-03T16:34:44.9368359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9368423Z outputs = self.model( 2025-11-03T16:34:44.9368684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9368753Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9368993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9369058Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9369265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9369345Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9369568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9369639Z return func(*args, **kwargs) 2025-11-03T16:34:44.9369880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9369977Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9370207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9370274Z return func(*args, **kwargs) 2025-11-03T16:34:44.9370515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9370609Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9370613Z 2025-11-03T16:34:44.9370715Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9370902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9370960Z res = mod(**inputs) 2025-11-03T16:34:44.9371212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9371276Z outputs = self.model( 2025-11-03T16:34:44.9371524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9371610Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9371844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9371920Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9372127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9372208Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9372429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9372493Z return func(*args, **kwargs) 2025-11-03T16:34:44.9372731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9372845Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9372850Z 2025-11-03T16:34:44.9372952Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9373134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9373203Z res = mod(**inputs) 2025-11-03T16:34:44.9373455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9373521Z outputs = self.model( 2025-11-03T16:34:44.9373766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9373831Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9374091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9374160Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9374372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9374457Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9374680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9374754Z return func(*args, **kwargs) 2025-11-03T16:34:44.9374987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9375105Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9375109Z 2025-11-03T16:34:44.9375203Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9375384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9375454Z res = mod(**inputs) 2025-11-03T16:34:44.9375688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9375758Z outputs = self.model( 2025-11-03T16:34:44.9375991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9376074Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9376322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9376388Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9376600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9376672Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9376897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9376966Z return func(*args, **kwargs) 2025-11-03T16:34:44.9377204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9377306Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9377310Z 2025-11-03T16:34:44.9377404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9377597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9377658Z res = mod(**inputs) 2025-11-03T16:34:44.9377894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9377965Z outputs = self.model( 2025-11-03T16:34:44.9378202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9378277Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9378519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9378589Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9378802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9378876Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9379125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9379191Z return func(*args, **kwargs) 2025-11-03T16:34:44.9379424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9379522Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9379760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9379834Z return func(*args, **kwargs) 2025-11-03T16:34:44.9380068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9380214Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9380219Z 2025-11-03T16:34:44.9380315Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9380500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9380566Z res = mod(**inputs) 2025-11-03T16:34:44.9380803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9380872Z outputs = self.model( 2025-11-03T16:34:44.9381113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9381180Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9381420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9381486Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9381696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9381786Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9382015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9382077Z return func(*args, **kwargs) 2025-11-03T16:34:44.9382308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9382406Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9382628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9382811Z return func(*args, **kwargs) 2025-11-03T16:34:44.9383065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9383148Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9383153Z 2025-11-03T16:34:44.9383259Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9383446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9383513Z res = mod(**inputs) 2025-11-03T16:34:44.9383760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9383834Z outputs = self.model( 2025-11-03T16:34:44.9384091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9384162Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9384413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9384479Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9384700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9384788Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9385011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9385083Z return func(*args, **kwargs) 2025-11-03T16:34:44.9385316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9385430Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9385654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9385720Z return func(*args, **kwargs) 2025-11-03T16:34:44.9385959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9386042Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9386046Z 2025-11-03T16:34:44.9386127Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9386222Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9386411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9386470Z res = mod(**inputs) 2025-11-03T16:34:44.9386713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9386784Z outputs = self.model( 2025-11-03T16:34:44.9387021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9387099Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9387340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9387431Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9387647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9387719Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9387949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9388012Z return func(*args, **kwargs) 2025-11-03T16:34:44.9388253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9388352Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9388597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9388671Z return func(*args, **kwargs) 2025-11-03T16:34:44.9388903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9389001Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9389269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9389395Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9389399Z 2025-11-03T16:34:44.9389501Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9389682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9389747Z res = mod(**inputs) 2025-11-03T16:34:44.9389986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9390048Z outputs = self.model( 2025-11-03T16:34:44.9390289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9390371Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9390617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9390683Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9390895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9390982Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9391208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9391281Z return func(*args, **kwargs) 2025-11-03T16:34:44.9391514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9391614Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9391838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9391902Z return func(*args, **kwargs) 2025-11-03T16:34:44.9392159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9392237Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9392240Z 2025-11-03T16:34:44.9392346Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9392535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9392606Z res = mod(**inputs) 2025-11-03T16:34:44.9392849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9392914Z outputs = self.model( 2025-11-03T16:34:44.9393181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9393250Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9393498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9393565Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9393778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9393860Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9394189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9394295Z return func(*args, **kwargs) 2025-11-03T16:34:44.9394549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 485, in forward 2025-11-03T16:34:44.9394632Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9394642Z 2025-11-03T16:34:44.9394744Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9394938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9395010Z res = mod(**inputs) 2025-11-03T16:34:44.9395268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9395343Z outputs = self.model( 2025-11-03T16:34:44.9395598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9395671Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9395936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9396007Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9396256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9396333Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9396605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9396676Z return func(*args, **kwargs) 2025-11-03T16:34:44.9396924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9397033Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9397255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9397326Z return func(*args, **kwargs) 2025-11-03T16:34:44.9397561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9397702Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9397706Z 2025-11-03T16:34:44.9397807Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9397991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9398056Z res = mod(**inputs) 2025-11-03T16:34:44.9398295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9398357Z outputs = self.model( 2025-11-03T16:34:44.9398599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9398667Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9398908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9398992Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9399198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9399276Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9399498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9399569Z return func(*args, **kwargs) 2025-11-03T16:34:44.9399801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9399909Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9400148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9400214Z return func(*args, **kwargs) 2025-11-03T16:34:44.9400456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9400530Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9400533Z 2025-11-03T16:34:44.9400636Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9400817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9400876Z res = mod(**inputs) 2025-11-03T16:34:44.9401122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9401187Z outputs = self.model( 2025-11-03T16:34:44.9401432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9401498Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9401739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9401821Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9402028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9402110Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9402333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9402404Z return func(*args, **kwargs) 2025-11-03T16:34:44.9402657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9402760Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9402993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9403057Z return func(*args, **kwargs) 2025-11-03T16:34:44.9403302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9403383Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9403387Z 2025-11-03T16:34:44.9403470Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9403565Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9403746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9403814Z res = mod(**inputs) 2025-11-03T16:34:44.9404053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9404125Z outputs = self.model( 2025-11-03T16:34:44.9404363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9404448Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9404694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9404760Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9404973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9405046Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9405272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9405344Z return func(*args, **kwargs) 2025-11-03T16:34:44.9405578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9405700Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9405922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9405989Z return func(*args, **kwargs) 2025-11-03T16:34:44.9406228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9406320Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9406602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9406729Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9406732Z 2025-11-03T16:34:44.9406833Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9407018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9407078Z res = mod(**inputs) 2025-11-03T16:34:44.9407324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9407407Z outputs = self.model( 2025-11-03T16:34:44.9407649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9407716Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9407954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9408041Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9408249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9408334Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9408561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9408633Z return func(*args, **kwargs) 2025-11-03T16:34:44.9408868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9408966Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9409198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9409263Z return func(*args, **kwargs) 2025-11-03T16:34:44.9409506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9409582Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9409585Z 2025-11-03T16:34:44.9409682Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9409869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9409930Z res = mod(**inputs) 2025-11-03T16:34:44.9410172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9410264Z outputs = self.model( 2025-11-03T16:34:44.9410508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9410574Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9410810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9410886Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9411092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9411186Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9411408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9411473Z return func(*args, **kwargs) 2025-11-03T16:34:44.9411714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9411825Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9411829Z 2025-11-03T16:34:44.9411931Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9412112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9412182Z res = mod(**inputs) 2025-11-03T16:34:44.9412416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9412480Z outputs = self.model( 2025-11-03T16:34:44.9412727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9412795Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9413061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9413130Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9413461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9413547Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9413817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9413890Z return func(*args, **kwargs) 2025-11-03T16:34:44.9414129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9414241Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9414254Z 2025-11-03T16:34:44.9414351Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9414537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9414604Z res = mod(**inputs) 2025-11-03T16:34:44.9414843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9414913Z outputs = self.model( 2025-11-03T16:34:44.9415150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9415218Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9415466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9415536Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9415752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9415855Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9416082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9416157Z return func(*args, **kwargs) 2025-11-03T16:34:44.9416393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9416480Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9416483Z 2025-11-03T16:34:44.9416584Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9416775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9416861Z res = mod(**inputs) 2025-11-03T16:34:44.9417096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9417167Z outputs = self.model( 2025-11-03T16:34:44.9417407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9417483Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9417718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9417784Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9417998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9418072Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9418307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9418373Z return func(*args, **kwargs) 2025-11-03T16:34:44.9418606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9418708Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9418957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9419030Z return func(*args, **kwargs) 2025-11-03T16:34:44.9419263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9419411Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9419431Z 2025-11-03T16:34:44.9419527Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9419708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9419778Z res = mod(**inputs) 2025-11-03T16:34:44.9420022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9420093Z outputs = self.model( 2025-11-03T16:34:44.9420337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9420404Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9420651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9420717Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9420937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9421010Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9421241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9421311Z return func(*args, **kwargs) 2025-11-03T16:34:44.9421550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9421668Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9421897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9421969Z return func(*args, **kwargs) 2025-11-03T16:34:44.9422202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9422279Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9422282Z 2025-11-03T16:34:44.9422385Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9422586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9422653Z res = mod(**inputs) 2025-11-03T16:34:44.9422890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9422954Z outputs = self.model( 2025-11-03T16:34:44.9423196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9423265Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9423512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9423579Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9423792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9423864Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9424088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9424160Z return func(*args, **kwargs) 2025-11-03T16:34:44.9424394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9424506Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9424727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9424790Z return func(*args, **kwargs) 2025-11-03T16:34:44.9425043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9425126Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9425129Z 2025-11-03T16:34:44.9425211Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9425308Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9425491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9425557Z res = mod(**inputs) 2025-11-03T16:34:44.9425798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9425868Z outputs = self.model( 2025-11-03T16:34:44.9426109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9426184Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9426425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9426493Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9426712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9426788Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9427020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9427102Z return func(*args, **kwargs) 2025-11-03T16:34:44.9427337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9427435Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9427658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9427728Z return func(*args, **kwargs) 2025-11-03T16:34:44.9427960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9428060Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9428347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9428472Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9428477Z 2025-11-03T16:34:44.9428578Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9428759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9428826Z res = mod(**inputs) 2025-11-03T16:34:44.9429063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9429125Z outputs = self.model( 2025-11-03T16:34:44.9429368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9429435Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9429677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9429746Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9429951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9430052Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9430274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9430346Z return func(*args, **kwargs) 2025-11-03T16:34:44.9430576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9430699Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9430924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9430990Z return func(*args, **kwargs) 2025-11-03T16:34:44.9431231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9431309Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9431312Z 2025-11-03T16:34:44.9431415Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9431601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9431660Z res = mod(**inputs) 2025-11-03T16:34:44.9431906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9431968Z outputs = self.model( 2025-11-03T16:34:44.9432211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9432280Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9432522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9432590Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9432818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9432900Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9433136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9433208Z return func(*args, **kwargs) 2025-11-03T16:34:44.9433456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9433558Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9433807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9433891Z return func(*args, **kwargs) 2025-11-03T16:34:44.9434208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9434376Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9434382Z 2025-11-03T16:34:44.9434492Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9434687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9434750Z res = mod(**inputs) 2025-11-03T16:34:44.9435013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9435082Z outputs = self.model( 2025-11-03T16:34:44.9435342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9435416Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9435673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9435755Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9436461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9436552Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9436795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9436862Z return func(*args, **kwargs) 2025-11-03T16:34:44.9437135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9437243Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9437490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9437559Z return func(*args, **kwargs) 2025-11-03T16:34:44.9437811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9437894Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9437898Z 2025-11-03T16:34:44.9438000Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9438201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9438265Z res = mod(**inputs) 2025-11-03T16:34:44.9438522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9438590Z outputs = self.model( 2025-11-03T16:34:44.9438838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9438922Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9439174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9439272Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9439490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9439572Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9439810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9439877Z return func(*args, **kwargs) 2025-11-03T16:34:44.9440136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9440257Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9440501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9440568Z return func(*args, **kwargs) 2025-11-03T16:34:44.9440814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9440906Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9440909Z 2025-11-03T16:34:44.9440988Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9441097Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9441289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9441353Z res = mod(**inputs) 2025-11-03T16:34:44.9441611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9441679Z outputs = self.model( 2025-11-03T16:34:44.9441938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9442008Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9442282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9442353Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9442571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9442657Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9442918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9442993Z return func(*args, **kwargs) 2025-11-03T16:34:44.9443240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9443346Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9443594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9443663Z return func(*args, **kwargs) 2025-11-03T16:34:44.9443917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9444014Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9444300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9444442Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9444446Z 2025-11-03T16:34:44.9444547Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9444749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9444811Z res = mod(**inputs) 2025-11-03T16:34:44.9445070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9445150Z outputs = self.model( 2025-11-03T16:34:44.9445387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9445463Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9445701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9445774Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9445984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9446072Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9446305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9446368Z return func(*args, **kwargs) 2025-11-03T16:34:44.9446613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9446711Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9446946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9447010Z return func(*args, **kwargs) 2025-11-03T16:34:44.9447247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9447335Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9447339Z 2025-11-03T16:34:44.9447436Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9447629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9447689Z res = mod(**inputs) 2025-11-03T16:34:44.9447926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9448018Z outputs = self.model( 2025-11-03T16:34:44.9448253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9448327Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9448560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9448647Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9448854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9448928Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9449160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9449223Z return func(*args, **kwargs) 2025-11-03T16:34:44.9449465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 502, in forward 2025-11-03T16:34:44.9449539Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9449543Z 2025-11-03T16:34:44.9449641Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9449830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9449889Z res = mod(**inputs) 2025-11-03T16:34:44.9450132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9450194Z outputs = self.model( 2025-11-03T16:34:44.9450431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9450505Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9450756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9450832Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9451037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9451114Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9451338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9451402Z return func(*args, **kwargs) 2025-11-03T16:34:44.9451640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9451769Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9451773Z 2025-11-03T16:34:44.9451872Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9452058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9452118Z res = mod(**inputs) 2025-11-03T16:34:44.9452360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9452423Z outputs = self.model( 2025-11-03T16:34:44.9452664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9452733Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9452975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9453043Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9453247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9453327Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9453580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9453651Z return func(*args, **kwargs) 2025-11-03T16:34:44.9453880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9453988Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9453992Z 2025-11-03T16:34:44.9454109Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9454294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9454360Z res = mod(**inputs) 2025-11-03T16:34:44.9454598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9454662Z outputs = self.model( 2025-11-03T16:34:44.9454909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9454979Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9455221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9455287Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9455502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9455576Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9455798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9455873Z return func(*args, **kwargs) 2025-11-03T16:34:44.9456107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9456208Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9456211Z 2025-11-03T16:34:44.9456308Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9456492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9456558Z res = mod(**inputs) 2025-11-03T16:34:44.9456803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9456874Z outputs = self.model( 2025-11-03T16:34:44.9457117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9457210Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9457447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9457516Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9457735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9457807Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9458046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9458112Z return func(*args, **kwargs) 2025-11-03T16:34:44.9458344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9458445Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9458667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9458740Z return func(*args, **kwargs) 2025-11-03T16:34:44.9458973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9459128Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9459138Z 2025-11-03T16:34:44.9459234Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9459415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9459482Z res = mod(**inputs) 2025-11-03T16:34:44.9459734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9459806Z outputs = self.model( 2025-11-03T16:34:44.9460043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9460112Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9460358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9460425Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9460649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9460723Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9460944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9461015Z return func(*args, **kwargs) 2025-11-03T16:34:44.9461250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9461348Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9461575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9461645Z return func(*args, **kwargs) 2025-11-03T16:34:44.9461879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9461972Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9461975Z 2025-11-03T16:34:44.9462079Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9462267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9462333Z res = mod(**inputs) 2025-11-03T16:34:44.9462579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9462641Z outputs = self.model( 2025-11-03T16:34:44.9462895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9462978Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9463220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9463290Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9463499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9463579Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9463803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9463873Z return func(*args, **kwargs) 2025-11-03T16:34:44.9464106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9464206Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9464427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9464490Z return func(*args, **kwargs) 2025-11-03T16:34:44.9464748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9464828Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9464832Z 2025-11-03T16:34:44.9464914Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9465010Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9465192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9465273Z res = mod(**inputs) 2025-11-03T16:34:44.9465509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9465580Z outputs = self.model( 2025-11-03T16:34:44.9465815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9465888Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9466124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9466190Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9466403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9466476Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9466711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9466774Z return func(*args, **kwargs) 2025-11-03T16:34:44.9467007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9467105Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9467329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9467416Z return func(*args, **kwargs) 2025-11-03T16:34:44.9467648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9467740Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9468020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9468146Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9468150Z 2025-11-03T16:34:44.9468251Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9468450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9468516Z res = mod(**inputs) 2025-11-03T16:34:44.9468765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9468829Z outputs = self.model( 2025-11-03T16:34:44.9469081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9469149Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9469398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9469466Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9469677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9469759Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9469989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9470063Z return func(*args, **kwargs) 2025-11-03T16:34:44.9470299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 476, in forward 2025-11-03T16:34:44.9470416Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:34:44.9470647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9470713Z return func(*args, **kwargs) 2025-11-03T16:34:44.9470960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9471056Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9471060Z 2025-11-03T16:34:44.9471166Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9471357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9471418Z res = mod(**inputs) 2025-11-03T16:34:44.9471670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9471738Z outputs = self.model( 2025-11-03T16:34:44.9471990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9472058Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9472303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9472380Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9472592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9472676Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9472908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9472982Z return func(*args, **kwargs) 2025-11-03T16:34:44.9473326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9473431Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9473683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9473750Z return func(*args, **kwargs) 2025-11-03T16:34:44.9474070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-11-03T16:34:44.9474222Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:34:44.9474255Z 2025-11-03T16:34:44.9474361Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9474549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9474609Z res = mod(**inputs) 2025-11-03T16:34:44.9474863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9474932Z outputs = self.model( 2025-11-03T16:34:44.9475194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9475264Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9475515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9475595Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9475813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9475924Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9476159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9476227Z return func(*args, **kwargs) 2025-11-03T16:34:44.9476494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9476597Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9476832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9476900Z return func(*args, **kwargs) 2025-11-03T16:34:44.9477162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 302, in forward 2025-11-03T16:34:44.9477241Z key_states = self.k_proj(current_states) 2025-11-03T16:34:44.9477245Z 2025-11-03T16:34:44.9477345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9477539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9477602Z res = mod(**inputs) 2025-11-03T16:34:44.9477852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9477918Z outputs = self.model( 2025-11-03T16:34:44.9478158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9478234Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9478477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9478552Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9478766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9478840Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9479076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9479161Z return func(*args, **kwargs) 2025-11-03T16:34:44.9479412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9479514Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9479754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9479822Z return func(*args, **kwargs) 2025-11-03T16:34:44.9480063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-11-03T16:34:44.9480171Z value_states = self.v_proj(current_states) 2025-11-03T16:34:44.9480174Z 2025-11-03T16:34:44.9480251Z cudagraph partition due to non gpu ops 2025-11-03T16:34:44.9480355Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9480546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9480608Z res = mod(**inputs) 2025-11-03T16:34:44.9480861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9480926Z outputs = self.model( 2025-11-03T16:34:44.9481174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9481243Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9481488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9481566Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9481779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9481863Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9482114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9482187Z return func(*args, **kwargs) 2025-11-03T16:34:44.9482431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9482534Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9482793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9482862Z return func(*args, **kwargs) 2025-11-03T16:34:44.9483118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 321, in forward 2025-11-03T16:34:44.9483218Z attn_output, attn_weights = attention_interface( 2025-11-03T16:34:44.9483507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:34:44.9483648Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:34:44.9483651Z 2025-11-03T16:34:44.9483753Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9483955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9484018Z res = mod(**inputs) 2025-11-03T16:34:44.9484291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9484355Z outputs = self.model( 2025-11-03T16:34:44.9484601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9484682Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9484927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9485022Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9485239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9485316Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9485561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9485627Z return func(*args, **kwargs) 2025-11-03T16:34:44.9485878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 493, in forward 2025-11-03T16:34:44.9485998Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-11-03T16:34:44.9486239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9486308Z return func(*args, **kwargs) 2025-11-03T16:34:44.9486553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 335, in forward 2025-11-03T16:34:44.9486639Z attn_output = self.out_proj(attn_output) 2025-11-03T16:34:44.9486642Z 2025-11-03T16:34:44.9486741Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9486936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9486996Z res = mod(**inputs) 2025-11-03T16:34:44.9487246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9487317Z outputs = self.model( 2025-11-03T16:34:44.9487563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9487640Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9487883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9487967Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9488189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9488265Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9488500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9488582Z return func(*args, **kwargs) 2025-11-03T16:34:44.9488834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9488948Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9488951Z 2025-11-03T16:34:44.9489046Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9489239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9489301Z res = mod(**inputs) 2025-11-03T16:34:44.9489549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9489613Z outputs = self.model( 2025-11-03T16:34:44.9489849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9489924Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9490161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9490236Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9490441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9490521Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9490765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9490828Z return func(*args, **kwargs) 2025-11-03T16:34:44.9491067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 507, in forward 2025-11-03T16:34:44.9491177Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:34:44.9491181Z 2025-11-03T16:34:44.9491281Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9491465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9491526Z res = mod(**inputs) 2025-11-03T16:34:44.9491788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9491851Z outputs = self.model( 2025-11-03T16:34:44.9492093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9492162Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9492399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9492475Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9492681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9492764Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9492985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9493056Z return func(*args, **kwargs) 2025-11-03T16:34:44.9493288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 509, in forward 2025-11-03T16:34:44.9493364Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:34:44.9493368Z 2025-11-03T16:34:44.9493485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9493667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9493732Z res = mod(**inputs) 2025-11-03T16:34:44.9493969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1402, in forward 2025-11-03T16:34:44.9494032Z outputs = self.model( 2025-11-03T16:34:44.9494288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1277, in forward 2025-11-03T16:34:44.9494357Z decoder_outputs = self.decoder( 2025-11-03T16:34:44.9494597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1131, in forward 2025-11-03T16:34:44.9494664Z layer_outputs = decoder_layer( 2025-11-03T16:34:44.9494877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:34:44.9494951Z return super().__call__(*args, **kwargs) 2025-11-03T16:34:44.9495173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:34:44.9495243Z return func(*args, **kwargs) 2025-11-03T16:34:44.9495477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 511, in forward 2025-11-03T16:34:44.9495560Z hidden_states = residual + hidden_states 2025-11-03T16:34:44.9495563Z 2025-11-03T16:34:44.9495657Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9495840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9495906Z res = mod(**inputs) 2025-11-03T16:34:44.9496143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1420, in forward 2025-11-03T16:34:44.9496256Z lm_logits = self.lm_head(outputs[0]) 2025-11-03T16:34:44.9496259Z 2025-11-03T16:34:44.9496354Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:34:44.9496538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:34:44.9496602Z res = mod(**inputs) 2025-11-03T16:34:44.9496844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1427, in forward 2025-11-03T16:34:44.9497008Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:34:44.9497027Z 2025-11-03T16:34:56.0755279Z Compilation time (from dynamo_timed): 25.661501204 2025-11-03T16:34:56.0844526Z pass 2025-11-03T16:34:56.0848669Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:34:56.0855043Z TIMING: _recursive_pre_grad_passes:0.01258 _recursive_joint_graph_passes:1.12881 _recursive_post_grad_passes:0.11758 async_compile.wait:0.66477 code_gen:10.60773 inductor_compile:13.49495 backend_compile:19.89311 gc:0.0001 entire_frame_compile:25.6615 total_wall_time:25.6615 2025-11-03T16:34:56.0862562Z STATS: call_* op count: 906 | FakeTensorMode.__torch_dispatch__:17859 | FakeTensor.__torch_dispatch__:10222 | ProxyTorchDispatchMode.__torch_dispatch__:4928 2025-11-03T16:34:56.0863068Z Dynamo produced 1 graphs covering 906 ops with 0 graph breaks (0 unique) 2025-11-03T16:34:58.6939789Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:34:58.6940841Z import pynvml # type: ignore[import] 2025-11-03T16:35:01.8164528Z 2025-11-03T16:35:04.0110115Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:35:04.0112106Z loading model: 0it [00:02, ?it/s] 2025-11-03T16:35:04.0128259Z cpu eval MBartForCausalLM 2025-11-03T16:35:05.6958045Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:06.3219856Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:06.9460884Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:13.7726635Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7731689Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7732070Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7732303Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7732499Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7732697Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7732887Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7733093Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7733304Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7733503Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7733692Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7733890Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7734127Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7734509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7734837Z res = mod(**inputs) 2025-11-03T16:35:13.7735231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7735625Z outputs = self.model.decoder( 2025-11-03T16:35:13.7736007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7736386Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7736937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7737300Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7737694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7738064Z return func(*args, **kwargs) 2025-11-03T16:35:13.7738445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7738854Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7739255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7739736Z return func(*args, **kwargs) 2025-11-03T16:35:13.7740110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.7740567Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.7740780Z 2025-11-03T16:35:13.7740888Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7741253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7741585Z res = mod(**inputs) 2025-11-03T16:35:13.7741956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7742345Z outputs = self.model.decoder( 2025-11-03T16:35:13.7742746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7743129Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7743468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7743833Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7744258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7744629Z return func(*args, **kwargs) 2025-11-03T16:35:13.7745002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7745411Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7745833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7746232Z return func(*args, **kwargs) 2025-11-03T16:35:13.7746610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.7747004Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.7747143Z 2025-11-03T16:35:13.7747259Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7747611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7747943Z res = mod(**inputs) 2025-11-03T16:35:13.7748318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7748711Z outputs = self.model.decoder( 2025-11-03T16:35:13.7749120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7749510Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7749868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7750242Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7750627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7751038Z return func(*args, **kwargs) 2025-11-03T16:35:13.7751414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7751881Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7752309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7752709Z return func(*args, **kwargs) 2025-11-03T16:35:13.7753100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.7753565Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.7753722Z 2025-11-03T16:35:13.7753811Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7754226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7754631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7754995Z res = mod(**inputs) 2025-11-03T16:35:13.7755382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7755768Z outputs = self.model.decoder( 2025-11-03T16:35:13.7756148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7756593Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7756944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7757310Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7757689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7758059Z return func(*args, **kwargs) 2025-11-03T16:35:13.7758445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7758863Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7759252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7759616Z return func(*args, **kwargs) 2025-11-03T16:35:13.7760005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.7760402Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.7760841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.7761328Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.7761509Z 2025-11-03T16:35:13.7761617Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7761972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7762280Z res = mod(**inputs) 2025-11-03T16:35:13.7762644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7763043Z outputs = self.model.decoder( 2025-11-03T16:35:13.7763427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7763810Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7764167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7764538Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7764908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7765302Z return func(*args, **kwargs) 2025-11-03T16:35:13.7765665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7766067Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7766461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7766825Z return func(*args, **kwargs) 2025-11-03T16:35:13.7767194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.7767580Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.7767744Z 2025-11-03T16:35:13.7767847Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7768198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7768515Z res = mod(**inputs) 2025-11-03T16:35:13.7768878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7769260Z outputs = self.model.decoder( 2025-11-03T16:35:13.7769628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7770008Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7770350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7770709Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7771069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7771433Z return func(*args, **kwargs) 2025-11-03T16:35:13.7771801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7772226Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7772421Z 2025-11-03T16:35:13.7772525Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7772877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7773194Z res = mod(**inputs) 2025-11-03T16:35:13.7773549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7773942Z outputs = self.model.decoder( 2025-11-03T16:35:13.7774319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7774710Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7775059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7775414Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7775776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7776140Z return func(*args, **kwargs) 2025-11-03T16:35:13.7776509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7776929Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7777305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.7777643Z return self.act(input) 2025-11-03T16:35:13.7777757Z 2025-11-03T16:35:13.7777862Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7778215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7778531Z res = mod(**inputs) 2025-11-03T16:35:13.7778904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7779293Z outputs = self.model.decoder( 2025-11-03T16:35:13.7779669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7780056Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7780404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7780763Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7781141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7781531Z return func(*args, **kwargs) 2025-11-03T16:35:13.7781906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.7782298Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.7782448Z 2025-11-03T16:35:13.7782557Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7782928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7783260Z res = mod(**inputs) 2025-11-03T16:35:13.7783660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7784053Z outputs = self.model.decoder( 2025-11-03T16:35:13.7784445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7784853Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7785207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7785571Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7785970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7786335Z return func(*args, **kwargs) 2025-11-03T16:35:13.7786704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7787111Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7787514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7787882Z return func(*args, **kwargs) 2025-11-03T16:35:13.7788247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.7788708Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.7788912Z 2025-11-03T16:35:13.7789021Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7789372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7789686Z res = mod(**inputs) 2025-11-03T16:35:13.7790051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7790426Z outputs = self.model.decoder( 2025-11-03T16:35:13.7790789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7791154Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7791488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7791836Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7792200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7792590Z return func(*args, **kwargs) 2025-11-03T16:35:13.7792948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7793340Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7793721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7794189Z return func(*args, **kwargs) 2025-11-03T16:35:13.7794550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.7794956Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.7795127Z 2025-11-03T16:35:13.7795239Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7795612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7795945Z res = mod(**inputs) 2025-11-03T16:35:13.7796303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7796685Z outputs = self.model.decoder( 2025-11-03T16:35:13.7797061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7797443Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7797780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7798139Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7798511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7798883Z return func(*args, **kwargs) 2025-11-03T16:35:13.7799250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7799646Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7800051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7800412Z return func(*args, **kwargs) 2025-11-03T16:35:13.7800772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.7801161Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.7801299Z 2025-11-03T16:35:13.7801399Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7801631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7801976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7802282Z res = mod(**inputs) 2025-11-03T16:35:13.7802623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7803001Z outputs = self.model.decoder( 2025-11-03T16:35:13.7803372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7803788Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7804131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7804482Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7804859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7805226Z return func(*args, **kwargs) 2025-11-03T16:35:13.7805605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7805990Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7806391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7806756Z return func(*args, **kwargs) 2025-11-03T16:35:13.7807113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.7807513Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.7807934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.7808399Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.7808581Z 2025-11-03T16:35:13.7808702Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7809054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7809368Z res = mod(**inputs) 2025-11-03T16:35:13.7809712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7810098Z outputs = self.model.decoder( 2025-11-03T16:35:13.7810471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7810856Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7811200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7811567Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7811946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7812326Z return func(*args, **kwargs) 2025-11-03T16:35:13.7812704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7813111Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7813727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7814094Z return func(*args, **kwargs) 2025-11-03T16:35:13.7814456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.7814847Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.7814984Z 2025-11-03T16:35:13.7815119Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7815471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7815789Z res = mod(**inputs) 2025-11-03T16:35:13.7816146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7816523Z outputs = self.model.decoder( 2025-11-03T16:35:13.7816902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7817281Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7817627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7817982Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7818350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7818725Z return func(*args, **kwargs) 2025-11-03T16:35:13.7819096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7819527Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7819703Z 2025-11-03T16:35:13.7819812Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7820193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7820518Z res = mod(**inputs) 2025-11-03T16:35:13.7820879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7821311Z outputs = self.model.decoder( 2025-11-03T16:35:13.7821681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7822073Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7822427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7822823Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7823207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7823574Z return func(*args, **kwargs) 2025-11-03T16:35:13.7823961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7824372Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7824742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.7825075Z return self.act(input) 2025-11-03T16:35:13.7825183Z 2025-11-03T16:35:13.7825283Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7825626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7825931Z res = mod(**inputs) 2025-11-03T16:35:13.7826273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7826632Z outputs = self.model.decoder( 2025-11-03T16:35:13.7826991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7827390Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7827949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7828284Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7828638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7829064Z return func(*args, **kwargs) 2025-11-03T16:35:13.7829429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.7829813Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.7829947Z 2025-11-03T16:35:13.7830052Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7830387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7830696Z res = mod(**inputs) 2025-11-03T16:35:13.7831041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7831413Z outputs = self.model.decoder( 2025-11-03T16:35:13.7831770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7832139Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7832474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7832821Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7833186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7833536Z return func(*args, **kwargs) 2025-11-03T16:35:13.7833896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-11-03T16:35:13.7834376Z hidden_states = residual + hidden_states 2025-11-03T16:35:13.7834522Z 2025-11-03T16:35:13.7834636Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7835007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7835306Z res = mod(**inputs) 2025-11-03T16:35:13.7835664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7836029Z outputs = self.model.decoder( 2025-11-03T16:35:13.7836381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7836759Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7837095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7837457Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7837813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7838165Z return func(*args, **kwargs) 2025-11-03T16:35:13.7838508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7838894Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7839270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7839618Z return func(*args, **kwargs) 2025-11-03T16:35:13.7839956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.7840386Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.7840585Z 2025-11-03T16:35:13.7840698Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7841031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7841329Z res = mod(**inputs) 2025-11-03T16:35:13.7841663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7842031Z outputs = self.model.decoder( 2025-11-03T16:35:13.7843191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7843561Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7843894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7844229Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7844581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7844941Z return func(*args, **kwargs) 2025-11-03T16:35:13.7845293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7845671Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7846043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7846397Z return func(*args, **kwargs) 2025-11-03T16:35:13.7846748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.7847123Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.7847250Z 2025-11-03T16:35:13.7847353Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7847696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7848025Z res = mod(**inputs) 2025-11-03T16:35:13.7848380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7848745Z outputs = self.model.decoder( 2025-11-03T16:35:13.7849095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7849459Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7849789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7850142Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7850503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7850857Z return func(*args, **kwargs) 2025-11-03T16:35:13.7851213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7851598Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7851970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7852307Z return func(*args, **kwargs) 2025-11-03T16:35:13.7852657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.7853040Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.7853174Z 2025-11-03T16:35:13.7853260Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7853490Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7853824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7854144Z res = mod(**inputs) 2025-11-03T16:35:13.7854507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7854882Z outputs = self.model.decoder( 2025-11-03T16:35:13.7855240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7855614Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7855969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7856325Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7856693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7857048Z return func(*args, **kwargs) 2025-11-03T16:35:13.7857410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7857811Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7858195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7858555Z return func(*args, **kwargs) 2025-11-03T16:35:13.7858910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.7859312Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.7859746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.7860216Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.7860395Z 2025-11-03T16:35:13.7860496Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7860842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7861167Z res = mod(**inputs) 2025-11-03T16:35:13.7861518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7861889Z outputs = self.model.decoder( 2025-11-03T16:35:13.7862251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7862626Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7862962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7863321Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7863712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7864075Z return func(*args, **kwargs) 2025-11-03T16:35:13.7864443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7864851Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7865244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7865594Z return func(*args, **kwargs) 2025-11-03T16:35:13.7865951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.7866336Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.7866468Z 2025-11-03T16:35:13.7866578Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7866924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7867225Z res = mod(**inputs) 2025-11-03T16:35:13.7867576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7867948Z outputs = self.model.decoder( 2025-11-03T16:35:13.7868330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7868710Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7869045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7869396Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7869782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7870141Z return func(*args, **kwargs) 2025-11-03T16:35:13.7870504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7870930Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7871108Z 2025-11-03T16:35:13.7871206Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7871560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7871875Z res = mod(**inputs) 2025-11-03T16:35:13.7872229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7872617Z outputs = self.model.decoder( 2025-11-03T16:35:13.7873004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7873394Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7873738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7874183Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7874622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7875033Z return func(*args, **kwargs) 2025-11-03T16:35:13.7875417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7875849Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7876255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.7876615Z return self.act(input) 2025-11-03T16:35:13.7876727Z 2025-11-03T16:35:13.7876838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7877197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7877527Z res = mod(**inputs) 2025-11-03T16:35:13.7877884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7878268Z outputs = self.model.decoder( 2025-11-03T16:35:13.7878644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7879017Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7879360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7879718Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7880091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7880457Z return func(*args, **kwargs) 2025-11-03T16:35:13.7880819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.7881213Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.7881353Z 2025-11-03T16:35:13.7881455Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7881830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7882146Z res = mod(**inputs) 2025-11-03T16:35:13.7882496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7882893Z outputs = self.model.decoder( 2025-11-03T16:35:13.7883303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7883690Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7884027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7884389Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7884764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7885136Z return func(*args, **kwargs) 2025-11-03T16:35:13.7885504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7885905Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7886298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7886678Z return func(*args, **kwargs) 2025-11-03T16:35:13.7887050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.7887515Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.7887720Z 2025-11-03T16:35:13.7887823Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7888177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7888515Z res = mod(**inputs) 2025-11-03T16:35:13.7888871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7889245Z outputs = self.model.decoder( 2025-11-03T16:35:13.7889617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7890005Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7890348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7890693Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7891059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7891407Z return func(*args, **kwargs) 2025-11-03T16:35:13.7891760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7892150Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7892520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7892858Z return func(*args, **kwargs) 2025-11-03T16:35:13.7893207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.7893575Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.7893700Z 2025-11-03T16:35:13.7893804Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7894130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7894426Z res = mod(**inputs) 2025-11-03T16:35:13.7894761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7895125Z outputs = self.model.decoder( 2025-11-03T16:35:13.7895512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7895868Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7896196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7896540Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7896912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7897254Z return func(*args, **kwargs) 2025-11-03T16:35:13.7897600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7897986Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7898359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7898716Z return func(*args, **kwargs) 2025-11-03T16:35:13.7899057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.7899434Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.7899571Z 2025-11-03T16:35:13.7899647Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7899873Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7900206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7900501Z res = mod(**inputs) 2025-11-03T16:35:13.7900839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7901195Z outputs = self.model.decoder( 2025-11-03T16:35:13.7901566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7901917Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7902257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7902596Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7902944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7903289Z return func(*args, **kwargs) 2025-11-03T16:35:13.7903629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7904054Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7904440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7904813Z return func(*args, **kwargs) 2025-11-03T16:35:13.7905187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.7905587Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.7906025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.7906507Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.7906688Z 2025-11-03T16:35:13.7906801Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7907149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7907467Z res = mod(**inputs) 2025-11-03T16:35:13.7907823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7908208Z outputs = self.model.decoder( 2025-11-03T16:35:13.7908601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7908969Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7909309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7909659Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7910045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7910403Z return func(*args, **kwargs) 2025-11-03T16:35:13.7910761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7911167Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7911561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7911935Z return func(*args, **kwargs) 2025-11-03T16:35:13.7912296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.7912694Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.7912832Z 2025-11-03T16:35:13.7912937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7913434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7913767Z res = mod(**inputs) 2025-11-03T16:35:13.7914177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7914596Z outputs = self.model.decoder( 2025-11-03T16:35:13.7915036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7915464Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7915811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7916162Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7916547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7916952Z return func(*args, **kwargs) 2025-11-03T16:35:13.7917326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7917745Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7917946Z 2025-11-03T16:35:13.7918047Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7918392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7918708Z res = mod(**inputs) 2025-11-03T16:35:13.7919067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7919445Z outputs = self.model.decoder( 2025-11-03T16:35:13.7919809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7920181Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7920522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7920869Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7921224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7921582Z return func(*args, **kwargs) 2025-11-03T16:35:13.7921940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7922373Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7922790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.7923125Z return self.act(input) 2025-11-03T16:35:13.7923239Z 2025-11-03T16:35:13.7923339Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7923690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7924033Z res = mod(**inputs) 2025-11-03T16:35:13.7924385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7924775Z outputs = self.model.decoder( 2025-11-03T16:35:13.7925155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7925538Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7925881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7926243Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7926619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7926989Z return func(*args, **kwargs) 2025-11-03T16:35:13.7927363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.7927746Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.7927888Z 2025-11-03T16:35:13.7927991Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7928348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7928672Z res = mod(**inputs) 2025-11-03T16:35:13.7929056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7929444Z outputs = self.model.decoder( 2025-11-03T16:35:13.7929822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7930206Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7930552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7930907Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7931283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7931675Z return func(*args, **kwargs) 2025-11-03T16:35:13.7932049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-11-03T16:35:13.7932444Z hidden_states = residual + hidden_states 2025-11-03T16:35:13.7932581Z 2025-11-03T16:35:13.7932686Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7933045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7933374Z res = mod(**inputs) 2025-11-03T16:35:13.7933733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7934119Z outputs = self.model.decoder( 2025-11-03T16:35:13.7934492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7934877Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7935223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7935591Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7935980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7936362Z return func(*args, **kwargs) 2025-11-03T16:35:13.7936720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7937122Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7937530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7937896Z return func(*args, **kwargs) 2025-11-03T16:35:13.7938273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.7938717Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.7938913Z 2025-11-03T16:35:13.7939021Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7939370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7939672Z res = mod(**inputs) 2025-11-03T16:35:13.7940020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7940392Z outputs = self.model.decoder( 2025-11-03T16:35:13.7940757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7941124Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7941459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7941808Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7942174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7942556Z return func(*args, **kwargs) 2025-11-03T16:35:13.7942915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7943315Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7943713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7944083Z return func(*args, **kwargs) 2025-11-03T16:35:13.7944451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.7944834Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.7945008Z 2025-11-03T16:35:13.7945110Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7945466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7945785Z res = mod(**inputs) 2025-11-03T16:35:13.7946138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7946520Z outputs = self.model.decoder( 2025-11-03T16:35:13.7946893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7947278Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7947628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7947982Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7948356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7948732Z return func(*args, **kwargs) 2025-11-03T16:35:13.7949104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7949506Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7949919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7950288Z return func(*args, **kwargs) 2025-11-03T16:35:13.7950656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.7951054Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.7951193Z 2025-11-03T16:35:13.7951295Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7951537Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7951899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7952219Z res = mod(**inputs) 2025-11-03T16:35:13.7952576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7952959Z outputs = self.model.decoder( 2025-11-03T16:35:13.7953339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7953723Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7954144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7954508Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7954894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7955274Z return func(*args, **kwargs) 2025-11-03T16:35:13.7955657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7956059Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7956442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7956837Z return func(*args, **kwargs) 2025-11-03T16:35:13.7957204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.7957610Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.7958046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.7958512Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.7958700Z 2025-11-03T16:35:13.7958827Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7959184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7959510Z res = mod(**inputs) 2025-11-03T16:35:13.7959864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7960261Z outputs = self.model.decoder( 2025-11-03T16:35:13.7960639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7961029Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7961387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7961750Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7962137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7962515Z return func(*args, **kwargs) 2025-11-03T16:35:13.7962889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7963303Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7963720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7964094Z return func(*args, **kwargs) 2025-11-03T16:35:13.7964463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.7964857Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.7964992Z 2025-11-03T16:35:13.7965113Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7965474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7965796Z res = mod(**inputs) 2025-11-03T16:35:13.7966157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7966546Z outputs = self.model.decoder( 2025-11-03T16:35:13.7966920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7967304Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7967656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7968019Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7968394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7968764Z return func(*args, **kwargs) 2025-11-03T16:35:13.7969135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7969565Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7969736Z 2025-11-03T16:35:13.7969846Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7970205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7970521Z res = mod(**inputs) 2025-11-03T16:35:13.7970875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7971260Z outputs = self.model.decoder( 2025-11-03T16:35:13.7971629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7971999Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7972342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7972722Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7973106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7973484Z return func(*args, **kwargs) 2025-11-03T16:35:13.7973863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.7974295Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.7974688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.7975032Z return self.act(input) 2025-11-03T16:35:13.7975147Z 2025-11-03T16:35:13.7975254Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7975617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7975939Z res = mod(**inputs) 2025-11-03T16:35:13.7976306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7976693Z outputs = self.model.decoder( 2025-11-03T16:35:13.7977064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7977476Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7977822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7978181Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7978541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7978954Z return func(*args, **kwargs) 2025-11-03T16:35:13.7979327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.7979721Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.7979855Z 2025-11-03T16:35:13.7979964Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7980309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7980629Z res = mod(**inputs) 2025-11-03T16:35:13.7980983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7981369Z outputs = self.model.decoder( 2025-11-03T16:35:13.7981748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7982124Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7982480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7982860Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7983283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7983718Z return func(*args, **kwargs) 2025-11-03T16:35:13.7984087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7984518Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7984910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7985273Z return func(*args, **kwargs) 2025-11-03T16:35:13.7985635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.7986097Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.7986306Z 2025-11-03T16:35:13.7986408Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7986794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7987111Z res = mod(**inputs) 2025-11-03T16:35:13.7987454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7987840Z outputs = self.model.decoder( 2025-11-03T16:35:13.7988223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7988609Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7988947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7989313Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7989710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7990096Z return func(*args, **kwargs) 2025-11-03T16:35:13.7990475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7990883Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7991299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7991666Z return func(*args, **kwargs) 2025-11-03T16:35:13.7992032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.7992418Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.7992549Z 2025-11-03T16:35:13.7992668Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7993024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7993348Z res = mod(**inputs) 2025-11-03T16:35:13.7993709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.7994205Z outputs = self.model.decoder( 2025-11-03T16:35:13.7994618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.7995018Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.7995371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.7995717Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.7996075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7996435Z return func(*args, **kwargs) 2025-11-03T16:35:13.7996799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.7997194Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.7997576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.7997967Z return func(*args, **kwargs) 2025-11-03T16:35:13.7998325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.7998709Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.7998845Z 2025-11-03T16:35:13.7998931Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.7999158Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.7999506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.7999818Z res = mod(**inputs) 2025-11-03T16:35:13.8000167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8000558Z outputs = self.model.decoder( 2025-11-03T16:35:13.8000920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8001301Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8001641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8001989Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8002345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8002703Z return func(*args, **kwargs) 2025-11-03T16:35:13.8003064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8003459Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8003842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8004193Z return func(*args, **kwargs) 2025-11-03T16:35:13.8004550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8004967Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8005407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8005892Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8006070Z 2025-11-03T16:35:13.8006176Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8006556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8006868Z res = mod(**inputs) 2025-11-03T16:35:13.8007218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8007590Z outputs = self.model.decoder( 2025-11-03T16:35:13.8007952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8008326Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8008660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8009010Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8009368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8009727Z return func(*args, **kwargs) 2025-11-03T16:35:13.8010087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8010480Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8010859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8011209Z return func(*args, **kwargs) 2025-11-03T16:35:13.8011595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8011983Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8012119Z 2025-11-03T16:35:13.8012232Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8012580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8012895Z res = mod(**inputs) 2025-11-03T16:35:13.8013422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8013830Z outputs = self.model.decoder( 2025-11-03T16:35:13.8014254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8014640Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8014990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8015366Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8015757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8016123Z return func(*args, **kwargs) 2025-11-03T16:35:13.8016482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8016905Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8017088Z 2025-11-03T16:35:13.8017194Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8017559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8017876Z res = mod(**inputs) 2025-11-03T16:35:13.8018240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8018619Z outputs = self.model.decoder( 2025-11-03T16:35:13.8019014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8019387Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8019721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8020096Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8020501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8020879Z return func(*args, **kwargs) 2025-11-03T16:35:13.8021254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8021662Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8022051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8022392Z return self.act(input) 2025-11-03T16:35:13.8022500Z 2025-11-03T16:35:13.8022611Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8022967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8023289Z res = mod(**inputs) 2025-11-03T16:35:13.8023647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8024032Z outputs = self.model.decoder( 2025-11-03T16:35:13.8024411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8024788Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8025135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8025525Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8025896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8026252Z return func(*args, **kwargs) 2025-11-03T16:35:13.8026628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8027016Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8027154Z 2025-11-03T16:35:13.8027261Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8027611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8027940Z res = mod(**inputs) 2025-11-03T16:35:13.8028294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8028675Z outputs = self.model.decoder( 2025-11-03T16:35:13.8029054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8029434Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8029769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8030134Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8030507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8030872Z return func(*args, **kwargs) 2025-11-03T16:35:13.8031236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-11-03T16:35:13.8031625Z hidden_states = residual + hidden_states 2025-11-03T16:35:13.8031763Z 2025-11-03T16:35:13.8031864Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8032237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8032558Z res = mod(**inputs) 2025-11-03T16:35:13.8032908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8033294Z outputs = self.model.decoder( 2025-11-03T16:35:13.8033686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8034128Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8034499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8034856Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8035237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8035620Z return func(*args, **kwargs) 2025-11-03T16:35:13.8035985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8036381Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8036771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8037140Z return func(*args, **kwargs) 2025-11-03T16:35:13.8037506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.8037961Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.8038166Z 2025-11-03T16:35:13.8038278Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8038613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8038934Z res = mod(**inputs) 2025-11-03T16:35:13.8039274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8039638Z outputs = self.model.decoder( 2025-11-03T16:35:13.8039984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8040342Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8040674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8041021Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8041401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8041756Z return func(*args, **kwargs) 2025-11-03T16:35:13.8042111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8042510Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8042890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8043235Z return func(*args, **kwargs) 2025-11-03T16:35:13.8043596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.8043980Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.8044109Z 2025-11-03T16:35:13.8044216Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8044564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8044864Z res = mod(**inputs) 2025-11-03T16:35:13.8045210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8045585Z outputs = self.model.decoder( 2025-11-03T16:35:13.8045993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8046357Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8046695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8047042Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8047441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8047800Z return func(*args, **kwargs) 2025-11-03T16:35:13.8048157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8048551Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8048939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8049302Z return func(*args, **kwargs) 2025-11-03T16:35:13.8049653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.8050052Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.8050196Z 2025-11-03T16:35:13.8050275Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.8050513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8050857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8051162Z res = mod(**inputs) 2025-11-03T16:35:13.8051511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8051885Z outputs = self.model.decoder( 2025-11-03T16:35:13.8052249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8052640Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8052972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8053325Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8053690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8054049Z return func(*args, **kwargs) 2025-11-03T16:35:13.8054397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8054813Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8055194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8055552Z return func(*args, **kwargs) 2025-11-03T16:35:13.8055912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8056299Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8056730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8057199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8057376Z 2025-11-03T16:35:13.8057485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8057826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8058132Z res = mod(**inputs) 2025-11-03T16:35:13.8058477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8058852Z outputs = self.model.decoder( 2025-11-03T16:35:13.8059239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8059607Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8059939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8060289Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8060676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8061032Z return func(*args, **kwargs) 2025-11-03T16:35:13.8061393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8061797Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8062189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8062558Z return func(*args, **kwargs) 2025-11-03T16:35:13.8062928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8063318Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8063461Z 2025-11-03T16:35:13.8063562Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8063917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8064228Z res = mod(**inputs) 2025-11-03T16:35:13.8064577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8064970Z outputs = self.model.decoder( 2025-11-03T16:35:13.8065347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8065740Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8066076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8066419Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8066783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8067146Z return func(*args, **kwargs) 2025-11-03T16:35:13.8067515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8067937Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8068128Z 2025-11-03T16:35:13.8068233Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8068590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8068908Z res = mod(**inputs) 2025-11-03T16:35:13.8069267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8069645Z outputs = self.model.decoder( 2025-11-03T16:35:13.8070021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8070404Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8070751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8071113Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8071483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8071858Z return func(*args, **kwargs) 2025-11-03T16:35:13.8072227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8072655Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8073064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8073401Z return self.act(input) 2025-11-03T16:35:13.8073516Z 2025-11-03T16:35:13.8073618Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8073973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8074383Z res = mod(**inputs) 2025-11-03T16:35:13.8074743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8075132Z outputs = self.model.decoder( 2025-11-03T16:35:13.8075506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8075893Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8076239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8076591Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8076968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8077339Z return func(*args, **kwargs) 2025-11-03T16:35:13.8077713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8078099Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8078238Z 2025-11-03T16:35:13.8078342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8078698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8079014Z res = mod(**inputs) 2025-11-03T16:35:13.8079369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8079762Z outputs = self.model.decoder( 2025-11-03T16:35:13.8080136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8080518Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8080863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8081228Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8081595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8081978Z return func(*args, **kwargs) 2025-11-03T16:35:13.8082345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8082755Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8083150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8083523Z return func(*args, **kwargs) 2025-11-03T16:35:13.8083894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.8084359Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.8084567Z 2025-11-03T16:35:13.8084680Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8085025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8085342Z res = mod(**inputs) 2025-11-03T16:35:13.8085700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8086090Z outputs = self.model.decoder( 2025-11-03T16:35:13.8086496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8086873Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8087219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8087576Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8087965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8088307Z return func(*args, **kwargs) 2025-11-03T16:35:13.8088658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8089050Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8089424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8089770Z return func(*args, **kwargs) 2025-11-03T16:35:13.8090113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.8090479Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.8090611Z 2025-11-03T16:35:13.8090707Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8091044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8091346Z res = mod(**inputs) 2025-11-03T16:35:13.8091676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8092043Z outputs = self.model.decoder( 2025-11-03T16:35:13.8092396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8092771Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8093092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8093431Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8093783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8094130Z return func(*args, **kwargs) 2025-11-03T16:35:13.8094479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8094854Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8095248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8095595Z return func(*args, **kwargs) 2025-11-03T16:35:13.8095944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.8096322Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.8096456Z 2025-11-03T16:35:13.8096533Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.8096757Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8097099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8097399Z res = mod(**inputs) 2025-11-03T16:35:13.8097728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8098090Z outputs = self.model.decoder( 2025-11-03T16:35:13.8098445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8098806Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8099133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8099496Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8099849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8100194Z return func(*args, **kwargs) 2025-11-03T16:35:13.8100540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8100934Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8101308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8101660Z return func(*args, **kwargs) 2025-11-03T16:35:13.8102014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8102404Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8102819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8103276Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8103279Z 2025-11-03T16:35:13.8103388Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8103576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8103640Z res = mod(**inputs) 2025-11-03T16:35:13.8103890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8103962Z outputs = self.model.decoder( 2025-11-03T16:35:13.8104208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8104277Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8104515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8104591Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8104816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8104890Z return func(*args, **kwargs) 2025-11-03T16:35:13.8105128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8105229Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8105453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8105546Z return func(*args, **kwargs) 2025-11-03T16:35:13.8105791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8105869Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8105874Z 2025-11-03T16:35:13.8105977Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8106161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8106228Z res = mod(**inputs) 2025-11-03T16:35:13.8106466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8106536Z outputs = self.model.decoder( 2025-11-03T16:35:13.8106777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8106845Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8107057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8107133Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8107375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8107445Z return func(*args, **kwargs) 2025-11-03T16:35:13.8107681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8107799Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8107803Z 2025-11-03T16:35:13.8107914Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8108101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8108171Z res = mod(**inputs) 2025-11-03T16:35:13.8108410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8108484Z outputs = self.model.decoder( 2025-11-03T16:35:13.8108726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8108798Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8109005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8109080Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8109313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8109378Z return func(*args, **kwargs) 2025-11-03T16:35:13.8109623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8109735Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8109934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8110024Z return self.act(input) 2025-11-03T16:35:13.8110028Z 2025-11-03T16:35:13.8110124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8110322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8110382Z res = mod(**inputs) 2025-11-03T16:35:13.8110627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8110704Z outputs = self.model.decoder( 2025-11-03T16:35:13.8110946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8111042Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8111247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8111327Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8111554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8111619Z return func(*args, **kwargs) 2025-11-03T16:35:13.8111865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8111940Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8111943Z 2025-11-03T16:35:13.8112048Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8112232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8112295Z res = mod(**inputs) 2025-11-03T16:35:13.8112542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8112609Z outputs = self.model.decoder( 2025-11-03T16:35:13.8112858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8112944Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8113168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8113406Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8113672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8113795Z return func(*args, **kwargs) 2025-11-03T16:35:13.8114097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-11-03T16:35:13.8114203Z hidden_states = residual + hidden_states 2025-11-03T16:35:13.8114207Z 2025-11-03T16:35:13.8114311Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8114509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8114586Z res = mod(**inputs) 2025-11-03T16:35:13.8114848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8114931Z outputs = self.model.decoder( 2025-11-03T16:35:13.8115189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8115259Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8115517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8115604Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8115847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8115913Z return func(*args, **kwargs) 2025-11-03T16:35:13.8116160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8116286Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8116517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8116592Z return func(*args, **kwargs) 2025-11-03T16:35:13.8116835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.8116988Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.8116992Z 2025-11-03T16:35:13.8117090Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8117304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8117376Z res = mod(**inputs) 2025-11-03T16:35:13.8117621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8117700Z outputs = self.model.decoder( 2025-11-03T16:35:13.8117946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8118018Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8118227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8118303Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8118538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8118605Z return func(*args, **kwargs) 2025-11-03T16:35:13.8118855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8118948Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8119202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8119277Z return func(*args, **kwargs) 2025-11-03T16:35:13.8119524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.8119608Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.8119611Z 2025-11-03T16:35:13.8119709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8119923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8119987Z res = mod(**inputs) 2025-11-03T16:35:13.8120234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8120309Z outputs = self.model.decoder( 2025-11-03T16:35:13.8120554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8120633Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8120845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8120919Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8121157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8121223Z return func(*args, **kwargs) 2025-11-03T16:35:13.8121475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8121569Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8121796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8121891Z return func(*args, **kwargs) 2025-11-03T16:35:13.8122137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.8122226Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.8122229Z 2025-11-03T16:35:13.8122310Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.8122414Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8122603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8122665Z res = mod(**inputs) 2025-11-03T16:35:13.8122914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8123011Z outputs = self.model.decoder( 2025-11-03T16:35:13.8123265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8123334Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8123548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8123629Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8123862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8123934Z return func(*args, **kwargs) 2025-11-03T16:35:13.8124180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8124273Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8124509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8124573Z return func(*args, **kwargs) 2025-11-03T16:35:13.8124828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8124944Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8125233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8125363Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8125367Z 2025-11-03T16:35:13.8125465Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8125679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8125740Z res = mod(**inputs) 2025-11-03T16:35:13.8125991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8126061Z outputs = self.model.decoder( 2025-11-03T16:35:13.8126304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8126383Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8126598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8126681Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8126915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8126980Z return func(*args, **kwargs) 2025-11-03T16:35:13.8127235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8127329Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8127567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8127632Z return func(*args, **kwargs) 2025-11-03T16:35:13.8127897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8127975Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8127978Z 2025-11-03T16:35:13.8128075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8128269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8128331Z res = mod(**inputs) 2025-11-03T16:35:13.8128585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8128652Z outputs = self.model.decoder( 2025-11-03T16:35:13.8128908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8128983Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8129191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8129274Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8129499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8129570Z return func(*args, **kwargs) 2025-11-03T16:35:13.8129806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8129918Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8129921Z 2025-11-03T16:35:13.8130022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8130207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8130275Z res = mod(**inputs) 2025-11-03T16:35:13.8130509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8130580Z outputs = self.model.decoder( 2025-11-03T16:35:13.8130840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8130908Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8131119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8131192Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8131433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8131505Z return func(*args, **kwargs) 2025-11-03T16:35:13.8131741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8131863Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8132064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8132136Z return self.act(input) 2025-11-03T16:35:13.8132140Z 2025-11-03T16:35:13.8132237Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8132419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8132488Z res = mod(**inputs) 2025-11-03T16:35:13.8132728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8132802Z outputs = self.model.decoder( 2025-11-03T16:35:13.8133038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8133105Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8133321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8133422Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8133652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8133716Z return func(*args, **kwargs) 2025-11-03T16:35:13.8133957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8134033Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8134038Z 2025-11-03T16:35:13.8134133Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8134323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8134399Z res = mod(**inputs) 2025-11-03T16:35:13.8134646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8134715Z outputs = self.model.decoder( 2025-11-03T16:35:13.8134955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8135028Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8135236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8135316Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8135547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8135612Z return func(*args, **kwargs) 2025-11-03T16:35:13.8135855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8135949Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8136181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8136263Z return func(*args, **kwargs) 2025-11-03T16:35:13.8136510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.8136654Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.8136658Z 2025-11-03T16:35:13.8136759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8136969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8137028Z res = mod(**inputs) 2025-11-03T16:35:13.8137272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8137342Z outputs = self.model.decoder( 2025-11-03T16:35:13.8137574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8137648Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8137856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8137936Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8138160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8138232Z return func(*args, **kwargs) 2025-11-03T16:35:13.8138468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8138561Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8138793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8138855Z return func(*args, **kwargs) 2025-11-03T16:35:13.8139122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.8139198Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.8139201Z 2025-11-03T16:35:13.8139295Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8139484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8139544Z res = mod(**inputs) 2025-11-03T16:35:13.8139785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8139854Z outputs = self.model.decoder( 2025-11-03T16:35:13.8140104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8140177Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8140382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8140466Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8140688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8140760Z return func(*args, **kwargs) 2025-11-03T16:35:13.8140993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8141084Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8141316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8141381Z return func(*args, **kwargs) 2025-11-03T16:35:13.8141622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.8141703Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.8141707Z 2025-11-03T16:35:13.8141783Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.8141899Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8142085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8142152Z res = mod(**inputs) 2025-11-03T16:35:13.8142390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8142475Z outputs = self.model.decoder( 2025-11-03T16:35:13.8142718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8142786Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8142999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8143070Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8143302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8143366Z return func(*args, **kwargs) 2025-11-03T16:35:13.8143596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8143695Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8143921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8143993Z return func(*args, **kwargs) 2025-11-03T16:35:13.8144224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8144316Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8144594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8144748Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8144751Z 2025-11-03T16:35:13.8144853Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8145038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8145108Z res = mod(**inputs) 2025-11-03T16:35:13.8145351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8145422Z outputs = self.model.decoder( 2025-11-03T16:35:13.8145667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8145748Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8145964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8146040Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8146268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8146340Z return func(*args, **kwargs) 2025-11-03T16:35:13.8146574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8146673Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8146897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8146970Z return func(*args, **kwargs) 2025-11-03T16:35:13.8147201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8147277Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8147282Z 2025-11-03T16:35:13.8147384Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8147585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8147651Z res = mod(**inputs) 2025-11-03T16:35:13.8147887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8147956Z outputs = self.model.decoder( 2025-11-03T16:35:13.8148214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8148282Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8148501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8148574Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8148798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8148868Z return func(*args, **kwargs) 2025-11-03T16:35:13.8149103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8149220Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8149224Z 2025-11-03T16:35:13.8149317Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8149511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8149570Z res = mod(**inputs) 2025-11-03T16:35:13.8149810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8149887Z outputs = self.model.decoder( 2025-11-03T16:35:13.8150124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8150218Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8150425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8150497Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8150730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8150796Z return func(*args, **kwargs) 2025-11-03T16:35:13.8151036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8151146Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8151370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8151435Z return self.act(input) 2025-11-03T16:35:13.8151439Z 2025-11-03T16:35:13.8151535Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8151726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8151785Z res = mod(**inputs) 2025-11-03T16:35:13.8152025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8152093Z outputs = self.model.decoder( 2025-11-03T16:35:13.8152327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8152402Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8152605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8152689Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8152923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8152992Z return func(*args, **kwargs) 2025-11-03T16:35:13.8153256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8153345Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8153348Z 2025-11-03T16:35:13.8153454Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8153638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8153722Z res = mod(**inputs) 2025-11-03T16:35:13.8153969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8154106Z outputs = self.model.decoder( 2025-11-03T16:35:13.8154367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8154436Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8154662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8154738Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8154974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8155053Z return func(*args, **kwargs) 2025-11-03T16:35:13.8155308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-11-03T16:35:13.8155397Z hidden_states = residual + hidden_states 2025-11-03T16:35:13.8155401Z 2025-11-03T16:35:13.8155504Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8155701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8155775Z res = mod(**inputs) 2025-11-03T16:35:13.8156029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8156131Z outputs = self.model.decoder( 2025-11-03T16:35:13.8156389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8156471Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8156683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8156760Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8156998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8157083Z return func(*args, **kwargs) 2025-11-03T16:35:13.8157333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8157429Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8157661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8157734Z return func(*args, **kwargs) 2025-11-03T16:35:13.8157977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.8158126Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.8158130Z 2025-11-03T16:35:13.8158228Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8158420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8158482Z res = mod(**inputs) 2025-11-03T16:35:13.8158726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8158803Z outputs = self.model.decoder( 2025-11-03T16:35:13.8159068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8159147Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8159357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8159430Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8159685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8159751Z return func(*args, **kwargs) 2025-11-03T16:35:13.8159999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8160095Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8160330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8160404Z return func(*args, **kwargs) 2025-11-03T16:35:13.8160650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.8160732Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.8160735Z 2025-11-03T16:35:13.8160834Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8161031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8161093Z res = mod(**inputs) 2025-11-03T16:35:13.8161340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8161419Z outputs = self.model.decoder( 2025-11-03T16:35:13.8161664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8161759Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8161974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8162048Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8162294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8162359Z return func(*args, **kwargs) 2025-11-03T16:35:13.8162615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8162707Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8162954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8163019Z return func(*args, **kwargs) 2025-11-03T16:35:13.8163254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.8163347Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.8163350Z 2025-11-03T16:35:13.8163427Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.8163530Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8163712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8163774Z res = mod(**inputs) 2025-11-03T16:35:13.8164023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8164090Z outputs = self.model.decoder( 2025-11-03T16:35:13.8164336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8164404Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8164612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8164711Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8164942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8165017Z return func(*args, **kwargs) 2025-11-03T16:35:13.8165262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8165384Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8165614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8165681Z return func(*args, **kwargs) 2025-11-03T16:35:13.8165927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8166021Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8166309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8166439Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8166442Z 2025-11-03T16:35:13.8166541Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8166737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8166801Z res = mod(**inputs) 2025-11-03T16:35:13.8167054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8167124Z outputs = self.model.decoder( 2025-11-03T16:35:13.8167375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8167445Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8167676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8167762Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8167997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8168068Z return func(*args, **kwargs) 2025-11-03T16:35:13.8168309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8168403Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8168644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8168736Z return func(*args, **kwargs) 2025-11-03T16:35:13.8168981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8169063Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8169068Z 2025-11-03T16:35:13.8169164Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8169361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8169421Z res = mod(**inputs) 2025-11-03T16:35:13.8169673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8169744Z outputs = self.model.decoder( 2025-11-03T16:35:13.8169993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8170061Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8170273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8170355Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8170606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8170680Z return func(*args, **kwargs) 2025-11-03T16:35:13.8170919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8171033Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8171036Z 2025-11-03T16:35:13.8171155Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8171342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8171413Z res = mod(**inputs) 2025-11-03T16:35:13.8171655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8171733Z outputs = self.model.decoder( 2025-11-03T16:35:13.8171973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8172041Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8172267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8172342Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8172581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8172648Z return func(*args, **kwargs) 2025-11-03T16:35:13.8172892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8173013Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8173216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8173307Z return self.act(input) 2025-11-03T16:35:13.8173310Z 2025-11-03T16:35:13.8173408Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8173594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8173662Z res = mod(**inputs) 2025-11-03T16:35:13.8173904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8173983Z outputs = self.model.decoder( 2025-11-03T16:35:13.8174227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8174319Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8174528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8174603Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8174842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8174905Z return func(*args, **kwargs) 2025-11-03T16:35:13.8175150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8175227Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8175230Z 2025-11-03T16:35:13.8175328Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8175522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8175582Z res = mod(**inputs) 2025-11-03T16:35:13.8175831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8175899Z outputs = self.model.decoder( 2025-11-03T16:35:13.8176139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8176232Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8176444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8176526Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8176755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8176826Z return func(*args, **kwargs) 2025-11-03T16:35:13.8177081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8177176Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8177414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8177480Z return func(*args, **kwargs) 2025-11-03T16:35:13.8177735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 227, in forward 2025-11-03T16:35:13.8177875Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:35:13.8177879Z 2025-11-03T16:35:13.8177976Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8178170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8178230Z res = mod(**inputs) 2025-11-03T16:35:13.8178480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8178550Z outputs = self.model.decoder( 2025-11-03T16:35:13.8178798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8178867Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8179096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8179180Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8179408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8179480Z return func(*args, **kwargs) 2025-11-03T16:35:13.8179718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8179813Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8180047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8180131Z return func(*args, **kwargs) 2025-11-03T16:35:13.8180379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 246, in forward 2025-11-03T16:35:13.8180459Z key_states = self.k_proj(current_states) 2025-11-03T16:35:13.8180462Z 2025-11-03T16:35:13.8180569Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8180754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8180815Z res = mod(**inputs) 2025-11-03T16:35:13.8181067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8181137Z outputs = self.model.decoder( 2025-11-03T16:35:13.8181387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8181457Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8181671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8181754Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8182009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8182083Z return func(*args, **kwargs) 2025-11-03T16:35:13.8182328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8182425Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8182691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8182759Z return func(*args, **kwargs) 2025-11-03T16:35:13.8183016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 247, in forward 2025-11-03T16:35:13.8183107Z value_states = self.v_proj(current_states) 2025-11-03T16:35:13.8183110Z 2025-11-03T16:35:13.8183197Z cudagraph partition due to non gpu ops 2025-11-03T16:35:13.8183301Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8183493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8183563Z res = mod(**inputs) 2025-11-03T16:35:13.8183821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8183895Z outputs = self.model.decoder( 2025-11-03T16:35:13.8184139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8184206Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8184426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8184504Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8184741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8184829Z return func(*args, **kwargs) 2025-11-03T16:35:13.8185072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8185173Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8185400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8185473Z return func(*args, **kwargs) 2025-11-03T16:35:13.8185717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 265, in forward 2025-11-03T16:35:13.8185836Z attn_output, attn_weights = attention_interface( 2025-11-03T16:35:13.8186117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:35:13.8186246Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:35:13.8186251Z 2025-11-03T16:35:13.8186357Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8186546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8186614Z res = mod(**inputs) 2025-11-03T16:35:13.8186860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8186929Z outputs = self.model.decoder( 2025-11-03T16:35:13.8187185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8187262Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8187480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8187552Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8187787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8187881Z return func(*args, **kwargs) 2025-11-03T16:35:13.8188114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 418, in forward 2025-11-03T16:35:13.8188214Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:35:13.8188437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8188525Z return func(*args, **kwargs) 2025-11-03T16:35:13.8188762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 279, in forward 2025-11-03T16:35:13.8188839Z attn_output = self.out_proj(attn_output) 2025-11-03T16:35:13.8188842Z 2025-11-03T16:35:13.8188944Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8189126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8189196Z res = mod(**inputs) 2025-11-03T16:35:13.8189435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8189501Z outputs = self.model.decoder( 2025-11-03T16:35:13.8189744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8189812Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8190022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8190095Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8190326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8190390Z return func(*args, **kwargs) 2025-11-03T16:35:13.8190646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8190763Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8190766Z 2025-11-03T16:35:13.8190859Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8191046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8191104Z res = mod(**inputs) 2025-11-03T16:35:13.8191342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8191417Z outputs = self.model.decoder( 2025-11-03T16:35:13.8191670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8191744Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8191949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8192034Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8192255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8192320Z return func(*args, **kwargs) 2025-11-03T16:35:13.8192560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 449, in forward 2025-11-03T16:35:13.8192667Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:35:13.8192872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:35:13.8192938Z return self.act(input) 2025-11-03T16:35:13.8192941Z 2025-11-03T16:35:13.8193036Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8193226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8193287Z res = mod(**inputs) 2025-11-03T16:35:13.8193562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8193632Z outputs = self.model.decoder( 2025-11-03T16:35:13.8193880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8193960Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8194273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8194365Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8194604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8194679Z return func(*args, **kwargs) 2025-11-03T16:35:13.8194928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-11-03T16:35:13.8195015Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:35:13.8195019Z 2025-11-03T16:35:13.8195129Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8195324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8195395Z res = mod(**inputs) 2025-11-03T16:35:13.8195648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-11-03T16:35:13.8195715Z outputs = self.model.decoder( 2025-11-03T16:35:13.8195961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-11-03T16:35:13.8196029Z layer_outputs = decoder_layer( 2025-11-03T16:35:13.8196241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:13.8196333Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:13.8196561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:13.8196639Z return func(*args, **kwargs) 2025-11-03T16:35:13.8196880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-11-03T16:35:13.8196970Z hidden_states = residual + hidden_states 2025-11-03T16:35:13.8196974Z 2025-11-03T16:35:13.8197071Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8197261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8197338Z res = mod(**inputs) 2025-11-03T16:35:13.8197571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1881, in forward 2025-11-03T16:35:13.8197654Z logits = self.lm_head(outputs[0]) 2025-11-03T16:35:13.8197657Z 2025-11-03T16:35:13.8197753Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:13.8197938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:13.8197998Z res = mod(**inputs) 2025-11-03T16:35:13.8198233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1887, in forward 2025-11-03T16:35:13.8198377Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:35:13.8198380Z 2025-11-03T16:35:23.5859074Z Compilation time (from dynamo_timed): 15.509406022 2025-11-03T16:35:23.6126074Z pass 2025-11-03T16:35:23.6128429Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:23.6130045Z TIMING: _recursive_pre_grad_passes:0.0065 _recursive_joint_graph_passes:0.62274 _recursive_post_grad_passes:0.06143 async_compile.wait:0.80212 code_gen:8.87877 inductor_compile:10.12257 backend_compile:13.11168 gc:0.00168 entire_frame_compile:15.50941 total_wall_time:15.50941 2025-11-03T16:35:23.6131281Z STATS: call_* op count: 337 | FakeTensorMode.__torch_dispatch__:7290 | FakeTensor.__torch_dispatch__:4449 | ProxyTorchDispatchMode.__torch_dispatch__:2023 2025-11-03T16:35:23.6131760Z Dynamo produced 1 graphs covering 337 ops with 0 graph breaks (0 unique) 2025-11-03T16:35:25.7982510Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:35:25.7983357Z import pynvml # type: ignore[import] 2025-11-03T16:35:28.9038117Z 2025-11-03T16:35:30.8727062Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:35:30.8732240Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:35:30.8742702Z cpu eval MT5ForConditionalGeneration 2025-11-03T16:35:31.4800420Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:31.7297606Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:31.9775552Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:43.1505411Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1509658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1514868Z res = mod(**inputs) 2025-11-03T16:35:43.1519923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1523888Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1528603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1533760Z layer_outputs = layer_module( 2025-11-03T16:35:43.1538570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1539035Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1539438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1539823Z return func(*args, **kwargs) 2025-11-03T16:35:43.1540200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1540641Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1541223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1541608Z return func(*args, **kwargs) 2025-11-03T16:35:43.1541988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1542383Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1542758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1543124Z return func(*args, **kwargs) 2025-11-03T16:35:43.1543478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 423, in forward 2025-11-03T16:35:43.1543871Z position_bias = position_bias + causal_mask 2025-11-03T16:35:43.1544021Z 2025-11-03T16:35:43.1544137Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1544506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1544840Z res = mod(**inputs) 2025-11-03T16:35:43.1545236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1545623Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1546134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1546501Z layer_outputs = layer_module( 2025-11-03T16:35:43.1546845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1547201Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1547637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1547999Z return func(*args, **kwargs) 2025-11-03T16:35:43.1548358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1548744Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1549134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1549516Z return func(*args, **kwargs) 2025-11-03T16:35:43.1549876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1550274Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1550685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1551083Z return func(*args, **kwargs) 2025-11-03T16:35:43.1551470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1551872Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1552026Z 2025-11-03T16:35:43.1552145Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1552539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1552926Z res = mod(**inputs) 2025-11-03T16:35:43.1553301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1553714Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1554284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1554720Z layer_outputs = layer_module( 2025-11-03T16:35:43.1555117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1555512Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1555928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1556307Z return func(*args, **kwargs) 2025-11-03T16:35:43.1556668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1557056Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1557428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1557799Z return func(*args, **kwargs) 2025-11-03T16:35:43.1558157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1558544Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1558917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1559291Z return func(*args, **kwargs) 2025-11-03T16:35:43.1559649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1560029Z key_states = self.k(current_states) 2025-11-03T16:35:43.1560166Z 2025-11-03T16:35:43.1560335Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1560700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1561027Z res = mod(**inputs) 2025-11-03T16:35:43.1561386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1561789Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1562209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1562627Z layer_outputs = layer_module( 2025-11-03T16:35:43.1562991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1563409Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1564013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1564537Z return func(*args, **kwargs) 2025-11-03T16:35:43.1564954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1565431Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1565903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1566343Z return func(*args, **kwargs) 2025-11-03T16:35:43.1566801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1567281Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1567848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1568319Z return func(*args, **kwargs) 2025-11-03T16:35:43.1569621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1570069Z value_states = self.v(current_states) 2025-11-03T16:35:43.1570234Z 2025-11-03T16:35:43.1570370Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1570703Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1586903Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1587330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1587674Z res = mod(**inputs) 2025-11-03T16:35:43.1588062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1588551Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1588961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1589363Z layer_outputs = layer_module( 2025-11-03T16:35:43.1589747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1590608Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1591009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1591400Z return func(*args, **kwargs) 2025-11-03T16:35:43.1591813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1592231Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1592639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1593040Z return func(*args, **kwargs) 2025-11-03T16:35:43.1593433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1593863Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1594337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1594740Z return func(*args, **kwargs) 2025-11-03T16:35:43.1595108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1595537Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1595676Z 2025-11-03T16:35:43.1595799Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1596168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1596502Z res = mod(**inputs) 2025-11-03T16:35:43.1596865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1597256Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1597641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1598018Z layer_outputs = layer_module( 2025-11-03T16:35:43.1598379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1598755Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1599145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1599522Z return func(*args, **kwargs) 2025-11-03T16:35:43.1599895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1600302Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1600703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1601182Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1601595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1602002Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1602165Z 2025-11-03T16:35:43.1602271Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1602639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1602968Z res = mod(**inputs) 2025-11-03T16:35:43.1603316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1603729Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1604128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1604559Z layer_outputs = layer_module( 2025-11-03T16:35:43.1604912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1605287Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1605717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1606099Z return func(*args, **kwargs) 2025-11-03T16:35:43.1606475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1606871Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1607275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1607702Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1608158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1608561Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1608702Z 2025-11-03T16:35:43.1608806Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1609173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1609510Z res = mod(**inputs) 2025-11-03T16:35:43.1609901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1610297Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1610694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1611101Z layer_outputs = layer_module( 2025-11-03T16:35:43.1611468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1611856Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1612233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1612614Z return func(*args, **kwargs) 2025-11-03T16:35:43.1612979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1613580Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1613985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1614404Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1614825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1615279Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1615423Z 2025-11-03T16:35:43.1615546Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1615897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1616218Z res = mod(**inputs) 2025-11-03T16:35:43.1616564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1616939Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1617316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1617713Z layer_outputs = layer_module( 2025-11-03T16:35:43.1618056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1618417Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1618792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1619160Z return func(*args, **kwargs) 2025-11-03T16:35:43.1619507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1619892Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1620278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1620693Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1621086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1621462Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1621605Z 2025-11-03T16:35:43.1621710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1622068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1622420Z res = mod(**inputs) 2025-11-03T16:35:43.1622761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1623130Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1623496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1623890Z layer_outputs = layer_module( 2025-11-03T16:35:43.1624232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1624590Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1624972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1625332Z return func(*args, **kwargs) 2025-11-03T16:35:43.1625681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1626042Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1626409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1626768Z return func(*args, **kwargs) 2025-11-03T16:35:43.1627112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1627478Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1627841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1628206Z return func(*args, **kwargs) 2025-11-03T16:35:43.1628560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1628953Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1629086Z 2025-11-03T16:35:43.1629199Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1629550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1629867Z res = mod(**inputs) 2025-11-03T16:35:43.1630215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1630593Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1630956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1631341Z layer_outputs = layer_module( 2025-11-03T16:35:43.1631684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1632045Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1632424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1632787Z return func(*args, **kwargs) 2025-11-03T16:35:43.1633146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1633530Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1633925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1634389Z return func(*args, **kwargs) 2025-11-03T16:35:43.1634781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1635188Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1635561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1635922Z return func(*args, **kwargs) 2025-11-03T16:35:43.1636283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1636648Z key_states = self.k(current_states) 2025-11-03T16:35:43.1636783Z 2025-11-03T16:35:43.1636883Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1637231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1637560Z res = mod(**inputs) 2025-11-03T16:35:43.1637897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1638273Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1638636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1639004Z layer_outputs = layer_module( 2025-11-03T16:35:43.1639339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1639697Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1640070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1640435Z return func(*args, **kwargs) 2025-11-03T16:35:43.1640786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1641168Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1641537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1641901Z return func(*args, **kwargs) 2025-11-03T16:35:43.1642236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1642614Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1642970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1643314Z return func(*args, **kwargs) 2025-11-03T16:35:43.1643648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1643991Z value_states = self.v(current_states) 2025-11-03T16:35:43.1644124Z 2025-11-03T16:35:43.1644203Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1644405Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1644629Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1644971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1645273Z res = mod(**inputs) 2025-11-03T16:35:43.1645598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1645951Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1646297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1646637Z layer_outputs = layer_module( 2025-11-03T16:35:43.1646963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1647302Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1647656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1647994Z return func(*args, **kwargs) 2025-11-03T16:35:43.1648328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1648684Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1649055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1649399Z return func(*args, **kwargs) 2025-11-03T16:35:43.1649731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1650090Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1650448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1650816Z return func(*args, **kwargs) 2025-11-03T16:35:43.1651156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1651501Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1651630Z 2025-11-03T16:35:43.1651728Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1652061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1652361Z res = mod(**inputs) 2025-11-03T16:35:43.1652681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1653028Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1653374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1653725Z layer_outputs = layer_module( 2025-11-03T16:35:43.1654048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1654378Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1654733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1655079Z return func(*args, **kwargs) 2025-11-03T16:35:43.1655429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1655799Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1656154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1656541Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1656924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1657296Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1657435Z 2025-11-03T16:35:43.1657531Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1657880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1658180Z res = mod(**inputs) 2025-11-03T16:35:43.1658511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1658867Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1659205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1659562Z layer_outputs = layer_module( 2025-11-03T16:35:43.1659897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1660246Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1660611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1660963Z return func(*args, **kwargs) 2025-11-03T16:35:43.1661311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1661690Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1662082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1662479Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1662866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1663225Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1663353Z 2025-11-03T16:35:43.1663474Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1663809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1664106Z res = mod(**inputs) 2025-11-03T16:35:43.1664435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1664791Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1665141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1665487Z layer_outputs = layer_module( 2025-11-03T16:35:43.1665807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1666145Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1666499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1666847Z return func(*args, **kwargs) 2025-11-03T16:35:43.1667177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1667542Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1667902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1668304Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1668686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1669043Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1669182Z 2025-11-03T16:35:43.1669278Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1669612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1669916Z res = mod(**inputs) 2025-11-03T16:35:43.1670235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1670602Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1670951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1671302Z layer_outputs = layer_module( 2025-11-03T16:35:43.1671628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1671962Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1672315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1672669Z return func(*args, **kwargs) 2025-11-03T16:35:43.1673013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1673388Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1673752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1674229Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1674643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1675031Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1675190Z 2025-11-03T16:35:43.1675306Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1675669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1675998Z res = mod(**inputs) 2025-11-03T16:35:43.1676360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1676735Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1677085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1677449Z layer_outputs = layer_module( 2025-11-03T16:35:43.1677781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1678133Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1678495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1678846Z return func(*args, **kwargs) 2025-11-03T16:35:43.1679195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1679560Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1679927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1680278Z return func(*args, **kwargs) 2025-11-03T16:35:43.1680628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1681003Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1681374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1681753Z return func(*args, **kwargs) 2025-11-03T16:35:43.1682098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1682468Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1682605Z 2025-11-03T16:35:43.1682705Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1683049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1683360Z res = mod(**inputs) 2025-11-03T16:35:43.1683688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1684064Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1684416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1684779Z layer_outputs = layer_module( 2025-11-03T16:35:43.1685106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1685454Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1685820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1686181Z return func(*args, **kwargs) 2025-11-03T16:35:43.1686533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1686895Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1687265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1687624Z return func(*args, **kwargs) 2025-11-03T16:35:43.1687972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1688336Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1688714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1689070Z return func(*args, **kwargs) 2025-11-03T16:35:43.1689425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1689792Z key_states = self.k(current_states) 2025-11-03T16:35:43.1689940Z 2025-11-03T16:35:43.1690049Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1690386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1690697Z res = mod(**inputs) 2025-11-03T16:35:43.1691035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1691403Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1691759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1692110Z layer_outputs = layer_module( 2025-11-03T16:35:43.1692445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1692789Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1693154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1693503Z return func(*args, **kwargs) 2025-11-03T16:35:43.1693858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1694215Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1694576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1694947Z return func(*args, **kwargs) 2025-11-03T16:35:43.1695290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1695663Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1696033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1696395Z return func(*args, **kwargs) 2025-11-03T16:35:43.1696758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1697115Z value_states = self.v(current_states) 2025-11-03T16:35:43.1697274Z 2025-11-03T16:35:43.1697352Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1697561Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1697795Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1698145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1698446Z res = mod(**inputs) 2025-11-03T16:35:43.1698782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1699146Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1699501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1699868Z layer_outputs = layer_module( 2025-11-03T16:35:43.1700194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1700542Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1700902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1701257Z return func(*args, **kwargs) 2025-11-03T16:35:43.1701623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1701993Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1702358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1702715Z return func(*args, **kwargs) 2025-11-03T16:35:43.1703084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1703451Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1703817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1704176Z return func(*args, **kwargs) 2025-11-03T16:35:43.1704522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1704884Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1705007Z 2025-11-03T16:35:43.1705109Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1705455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1705767Z res = mod(**inputs) 2025-11-03T16:35:43.1706091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1706441Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1706791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1707141Z layer_outputs = layer_module( 2025-11-03T16:35:43.1707464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1707806Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1708167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1708513Z return func(*args, **kwargs) 2025-11-03T16:35:43.1708847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1709212Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1709572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1709958Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1710345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1710732Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1710873Z 2025-11-03T16:35:43.1710978Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1711305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1711603Z res = mod(**inputs) 2025-11-03T16:35:43.1711928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1712280Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1712624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1712968Z layer_outputs = layer_module( 2025-11-03T16:35:43.1713425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1713833Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1714269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1714688Z return func(*args, **kwargs) 2025-11-03T16:35:43.1715131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1715516Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1715898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1716349Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1716750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1717110Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1717245Z 2025-11-03T16:35:43.1717342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1717680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1717987Z res = mod(**inputs) 2025-11-03T16:35:43.1718312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1718666Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1719017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1719369Z layer_outputs = layer_module( 2025-11-03T16:35:43.1719688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1720028Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1720386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1720734Z return func(*args, **kwargs) 2025-11-03T16:35:43.1721071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1721459Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1721826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1722229Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1722617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1722982Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1723115Z 2025-11-03T16:35:43.1723213Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1723548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1723875Z res = mod(**inputs) 2025-11-03T16:35:43.1724199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1724544Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1724894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1725239Z layer_outputs = layer_module( 2025-11-03T16:35:43.1725564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1725904Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1726254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1726604Z return func(*args, **kwargs) 2025-11-03T16:35:43.1726943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1727308Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1727673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1728067Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1728455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1728811Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1728936Z 2025-11-03T16:35:43.1729039Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1729385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1729690Z res = mod(**inputs) 2025-11-03T16:35:43.1730013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1730371Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1730718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1731062Z layer_outputs = layer_module( 2025-11-03T16:35:43.1731395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1731734Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1732086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1732434Z return func(*args, **kwargs) 2025-11-03T16:35:43.1732766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1733124Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1733484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1733827Z return func(*args, **kwargs) 2025-11-03T16:35:43.1734156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1734532Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1734887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1735230Z return func(*args, **kwargs) 2025-11-03T16:35:43.1735564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1735911Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1736043Z 2025-11-03T16:35:43.1736141Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1736497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1736805Z res = mod(**inputs) 2025-11-03T16:35:43.1737133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1737494Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1737853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1738211Z layer_outputs = layer_module( 2025-11-03T16:35:43.1738544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1738882Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1739245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1739597Z return func(*args, **kwargs) 2025-11-03T16:35:43.1739942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1740305Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1740658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1741027Z return func(*args, **kwargs) 2025-11-03T16:35:43.1741362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1741718Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1742065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1742431Z return func(*args, **kwargs) 2025-11-03T16:35:43.1742768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1743125Z key_states = self.k(current_states) 2025-11-03T16:35:43.1743250Z 2025-11-03T16:35:43.1743356Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1743693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1743993Z res = mod(**inputs) 2025-11-03T16:35:43.1744320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1744676Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1745013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1745361Z layer_outputs = layer_module( 2025-11-03T16:35:43.1745690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1746031Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1746388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1746725Z return func(*args, **kwargs) 2025-11-03T16:35:43.1747066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1747437Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1747793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1748136Z return func(*args, **kwargs) 2025-11-03T16:35:43.1748462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1748826Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1749181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1749552Z return func(*args, **kwargs) 2025-11-03T16:35:43.1749879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1750231Z value_states = self.v(current_states) 2025-11-03T16:35:43.1750365Z 2025-11-03T16:35:43.1750442Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1750641Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1750862Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1751191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1751490Z res = mod(**inputs) 2025-11-03T16:35:43.1751818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1752171Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1752508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1752860Z layer_outputs = layer_module( 2025-11-03T16:35:43.1753184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1753521Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1753902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1754336Z return func(*args, **kwargs) 2025-11-03T16:35:43.1754706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1755098Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1755492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1755873Z return func(*args, **kwargs) 2025-11-03T16:35:43.1756228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1756618Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1757002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1757375Z return func(*args, **kwargs) 2025-11-03T16:35:43.1757727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1758105Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1758244Z 2025-11-03T16:35:43.1758345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1758705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1759025Z res = mod(**inputs) 2025-11-03T16:35:43.1759365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1759738Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1760115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1760505Z layer_outputs = layer_module( 2025-11-03T16:35:43.1760843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1761209Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1761595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1761972Z return func(*args, **kwargs) 2025-11-03T16:35:43.1762336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1762710Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1763112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1763486Z return func(*args, **kwargs) 2025-11-03T16:35:43.1763886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-11-03T16:35:43.1764304Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.1764483Z 2025-11-03T16:35:43.1764582Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1764924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1765237Z res = mod(**inputs) 2025-11-03T16:35:43.1765573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1765926Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1766279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1766643Z layer_outputs = layer_module( 2025-11-03T16:35:43.1766980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1767331Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1767734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1768099Z return func(*args, **kwargs) 2025-11-03T16:35:43.1768446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1768823Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1769217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1769616Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1770014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1770394Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1770541Z 2025-11-03T16:35:43.1770647Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1770980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1771286Z res = mod(**inputs) 2025-11-03T16:35:43.1771619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1771980Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1772337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1772692Z layer_outputs = layer_module( 2025-11-03T16:35:43.1773027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1773375Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1773739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1774136Z return func(*args, **kwargs) 2025-11-03T16:35:43.1774498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1774893Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1775292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1775716Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1776106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1776490Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1776626Z 2025-11-03T16:35:43.1776727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1777073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1777383Z res = mod(**inputs) 2025-11-03T16:35:43.1777710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1778072Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1778428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1778785Z layer_outputs = layer_module( 2025-11-03T16:35:43.1779110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1779458Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1779824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1780183Z return func(*args, **kwargs) 2025-11-03T16:35:43.1780532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1780916Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1781293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1781691Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1782083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1782484Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1782624Z 2025-11-03T16:35:43.1782726Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1783082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1783398Z res = mod(**inputs) 2025-11-03T16:35:43.1783748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1784103Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1784459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1784817Z layer_outputs = layer_module( 2025-11-03T16:35:43.1785162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1785522Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1785886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1786250Z return func(*args, **kwargs) 2025-11-03T16:35:43.1786603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1786986Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1787397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1787800Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1788202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1788576Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1788710Z 2025-11-03T16:35:43.1788819Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1789174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1789482Z res = mod(**inputs) 2025-11-03T16:35:43.1789842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1790226Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1790589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1790954Z layer_outputs = layer_module( 2025-11-03T16:35:43.1791304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1791667Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1792039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1792410Z return func(*args, **kwargs) 2025-11-03T16:35:43.1792760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1793137Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1793520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1793867Z return func(*args, **kwargs) 2025-11-03T16:35:43.1794290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1794661Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1795027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1795394Z return func(*args, **kwargs) 2025-11-03T16:35:43.1795777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1796140Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1796281Z 2025-11-03T16:35:43.1796382Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1796732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1797050Z res = mod(**inputs) 2025-11-03T16:35:43.1797370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1797730Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1798079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1798430Z layer_outputs = layer_module( 2025-11-03T16:35:43.1798759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1799093Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1799451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1799802Z return func(*args, **kwargs) 2025-11-03T16:35:43.1800139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1800499Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1800880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1801227Z return func(*args, **kwargs) 2025-11-03T16:35:43.1801561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1801916Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1802267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1802613Z return func(*args, **kwargs) 2025-11-03T16:35:43.1802946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1803311Z key_states = self.k(current_states) 2025-11-03T16:35:43.1803433Z 2025-11-03T16:35:43.1803537Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1803864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1804163Z res = mod(**inputs) 2025-11-03T16:35:43.1804489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1804840Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1805180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1805522Z layer_outputs = layer_module( 2025-11-03T16:35:43.1805847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1806184Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1806535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1806872Z return func(*args, **kwargs) 2025-11-03T16:35:43.1807230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1807590Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1807946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1808290Z return func(*args, **kwargs) 2025-11-03T16:35:43.1808637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1808999Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1809355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1809705Z return func(*args, **kwargs) 2025-11-03T16:35:43.1810031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1810387Z value_states = self.v(current_states) 2025-11-03T16:35:43.1810521Z 2025-11-03T16:35:43.1810598Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1810802Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1811026Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1811351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1811655Z res = mod(**inputs) 2025-11-03T16:35:43.1811984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1812336Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1812678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1813040Z layer_outputs = layer_module( 2025-11-03T16:35:43.1813532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1813932Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1814302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1814656Z return func(*args, **kwargs) 2025-11-03T16:35:43.1815015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1815384Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1815766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1816140Z return func(*args, **kwargs) 2025-11-03T16:35:43.1816476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1816869Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1817243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1817600Z return func(*args, **kwargs) 2025-11-03T16:35:43.1817937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1818301Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1818431Z 2025-11-03T16:35:43.1818530Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1818882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1819193Z res = mod(**inputs) 2025-11-03T16:35:43.1819523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1819883Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1820239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1820601Z layer_outputs = layer_module( 2025-11-03T16:35:43.1820956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1821309Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1821674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1822031Z return func(*args, **kwargs) 2025-11-03T16:35:43.1822402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1822774Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1823154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1823555Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1823958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1824341Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1824485Z 2025-11-03T16:35:43.1824586Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1824928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1825241Z res = mod(**inputs) 2025-11-03T16:35:43.1825584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1825947Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1826300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1826659Z layer_outputs = layer_module( 2025-11-03T16:35:43.1826996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1827363Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1827722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1828083Z return func(*args, **kwargs) 2025-11-03T16:35:43.1828428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1828816Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1829180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1829577Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1829963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1830320Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1830446Z 2025-11-03T16:35:43.1830550Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1830880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1831172Z res = mod(**inputs) 2025-11-03T16:35:43.1831498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1831848Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1832196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1832539Z layer_outputs = layer_module( 2025-11-03T16:35:43.1832876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1833225Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1833602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1833964Z return func(*args, **kwargs) 2025-11-03T16:35:43.1834374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1834758Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1835208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1835610Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1836002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1836367Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1836508Z 2025-11-03T16:35:43.1836606Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1836945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1837248Z res = mod(**inputs) 2025-11-03T16:35:43.1837567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1837923Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1838275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1838632Z layer_outputs = layer_module( 2025-11-03T16:35:43.1838958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1839297Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1839658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1840009Z return func(*args, **kwargs) 2025-11-03T16:35:43.1840370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1840737Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1841110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1841503Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1841897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1842256Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1842401Z 2025-11-03T16:35:43.1842497Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1842835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1843134Z res = mod(**inputs) 2025-11-03T16:35:43.1843465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1843817Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1844158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1844506Z layer_outputs = layer_module( 2025-11-03T16:35:43.1844833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1845174Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1845520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1845874Z return func(*args, **kwargs) 2025-11-03T16:35:43.1846215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1846574Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1846951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1847292Z return func(*args, **kwargs) 2025-11-03T16:35:43.1847631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1847985Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1848362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1848708Z return func(*args, **kwargs) 2025-11-03T16:35:43.1849036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1849389Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1849514Z 2025-11-03T16:35:43.1849619Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1849955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1850250Z res = mod(**inputs) 2025-11-03T16:35:43.1850576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1850928Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1851275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1851624Z layer_outputs = layer_module( 2025-11-03T16:35:43.1851941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1852282Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1852634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1852992Z return func(*args, **kwargs) 2025-11-03T16:35:43.1853322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1853676Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1854030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1854380Z return func(*args, **kwargs) 2025-11-03T16:35:43.1854716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1855063Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1855422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1855786Z return func(*args, **kwargs) 2025-11-03T16:35:43.1856122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1856474Z key_states = self.k(current_states) 2025-11-03T16:35:43.1856597Z 2025-11-03T16:35:43.1856694Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1857029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1857330Z res = mod(**inputs) 2025-11-03T16:35:43.1857657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1858004Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1858353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1858702Z layer_outputs = layer_module( 2025-11-03T16:35:43.1859026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1859369Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1859729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1860082Z return func(*args, **kwargs) 2025-11-03T16:35:43.1860421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1860777Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1861142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1861491Z return func(*args, **kwargs) 2025-11-03T16:35:43.1861826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1862187Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1862538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1862879Z return func(*args, **kwargs) 2025-11-03T16:35:43.1863214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1863568Z value_states = self.v(current_states) 2025-11-03T16:35:43.1863691Z 2025-11-03T16:35:43.1863774Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1863974Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1864189Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1864525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1864825Z res = mod(**inputs) 2025-11-03T16:35:43.1865151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1865491Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1865839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1866207Z layer_outputs = layer_module( 2025-11-03T16:35:43.1866531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1866873Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1867221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1867569Z return func(*args, **kwargs) 2025-11-03T16:35:43.1867908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1868283Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1868632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1868979Z return func(*args, **kwargs) 2025-11-03T16:35:43.1869315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1869673Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1870029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1870369Z return func(*args, **kwargs) 2025-11-03T16:35:43.1870707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1871060Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1871180Z 2025-11-03T16:35:43.1871288Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1871627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1871924Z res = mod(**inputs) 2025-11-03T16:35:43.1872252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1872623Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1872975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1873321Z layer_outputs = layer_module( 2025-11-03T16:35:43.1873650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1874245Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1874659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1875037Z return func(*args, **kwargs) 2025-11-03T16:35:43.1875394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1875781Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1876163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1876526Z return func(*args, **kwargs) 2025-11-03T16:35:43.1876880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-11-03T16:35:43.1877314Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.1877505Z 2025-11-03T16:35:43.1877610Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1877969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1878293Z res = mod(**inputs) 2025-11-03T16:35:43.1878643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1879016Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1879403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1879775Z layer_outputs = layer_module( 2025-11-03T16:35:43.1880117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1880469Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1880843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1881211Z return func(*args, **kwargs) 2025-11-03T16:35:43.1881567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1881968Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1882351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1882760Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1883166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1883560Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1883707Z 2025-11-03T16:35:43.1883808Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1884157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1884474Z res = mod(**inputs) 2025-11-03T16:35:43.1884821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1885194Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1885552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1885916Z layer_outputs = layer_module( 2025-11-03T16:35:43.1886276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1886636Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1887003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1887372Z return func(*args, **kwargs) 2025-11-03T16:35:43.1887741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1888129Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1888517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1888923Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1889340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1889698Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1889824Z 2025-11-03T16:35:43.1889929Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1890268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1890562Z res = mod(**inputs) 2025-11-03T16:35:43.1890887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1891240Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1891590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1891932Z layer_outputs = layer_module( 2025-11-03T16:35:43.1892257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1892615Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1892976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1893332Z return func(*args, **kwargs) 2025-11-03T16:35:43.1893663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1894026Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1894391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1894779Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1895176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1895531Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1895671Z 2025-11-03T16:35:43.1895770Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1896104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1896404Z res = mod(**inputs) 2025-11-03T16:35:43.1896720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1897069Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1897413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1897759Z layer_outputs = layer_module( 2025-11-03T16:35:43.1898081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1898418Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1898766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1899115Z return func(*args, **kwargs) 2025-11-03T16:35:43.1899472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1899833Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1900199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1900589Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1900991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1901350Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1901475Z 2025-11-03T16:35:43.1901571Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1901915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1902218Z res = mod(**inputs) 2025-11-03T16:35:43.1902546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1902895Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1903233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1903581Z layer_outputs = layer_module( 2025-11-03T16:35:43.1903905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1904248Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1904594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1904942Z return func(*args, **kwargs) 2025-11-03T16:35:43.1905275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1905655Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1906012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1906354Z return func(*args, **kwargs) 2025-11-03T16:35:43.1906690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1907048Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1907404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1907768Z return func(*args, **kwargs) 2025-11-03T16:35:43.1908097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1908452Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1908586Z 2025-11-03T16:35:43.1908684Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1909024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1909318Z res = mod(**inputs) 2025-11-03T16:35:43.1909653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1910010Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1910362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1910714Z layer_outputs = layer_module( 2025-11-03T16:35:43.1911032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1911378Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1911733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1912082Z return func(*args, **kwargs) 2025-11-03T16:35:43.1912427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1912784Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1913145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1913659Z return func(*args, **kwargs) 2025-11-03T16:35:43.1914086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1914470Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1914851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1915217Z return func(*args, **kwargs) 2025-11-03T16:35:43.1915577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1915943Z key_states = self.k(current_states) 2025-11-03T16:35:43.1916071Z 2025-11-03T16:35:43.1916171Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1916531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1916844Z res = mod(**inputs) 2025-11-03T16:35:43.1917186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1917258Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1917493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1917572Z layer_outputs = layer_module( 2025-11-03T16:35:43.1917787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1917900Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1918133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1918209Z return func(*args, **kwargs) 2025-11-03T16:35:43.1918438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1918517Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1918755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1918820Z return func(*args, **kwargs) 2025-11-03T16:35:43.1919079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1919160Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1919389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1919463Z return func(*args, **kwargs) 2025-11-03T16:35:43.1919690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1919773Z value_states = self.v(current_states) 2025-11-03T16:35:43.1919777Z 2025-11-03T16:35:43.1919856Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1919940Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1920042Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1920235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1920306Z res = mod(**inputs) 2025-11-03T16:35:43.1920539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1920618Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1920877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1920947Z layer_outputs = layer_module( 2025-11-03T16:35:43.1921166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1921242Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1921498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1921564Z return func(*args, **kwargs) 2025-11-03T16:35:43.1921792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1921878Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1922109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1922182Z return func(*args, **kwargs) 2025-11-03T16:35:43.1922413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1922494Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1922731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1922796Z return func(*args, **kwargs) 2025-11-03T16:35:43.1923030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1923103Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1923108Z 2025-11-03T16:35:43.1923215Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1923402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1923482Z res = mod(**inputs) 2025-11-03T16:35:43.1923721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1923791Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1924028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1924097Z layer_outputs = layer_module( 2025-11-03T16:35:43.1924310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1924392Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1924620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1924709Z return func(*args, **kwargs) 2025-11-03T16:35:43.1924938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1925029Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1925265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1925378Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1925612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1925706Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1925711Z 2025-11-03T16:35:43.1925813Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1926003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1926065Z res = mod(**inputs) 2025-11-03T16:35:43.1926303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1926374Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1926626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1926696Z layer_outputs = layer_module( 2025-11-03T16:35:43.1926909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1926992Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1927294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1927367Z return func(*args, **kwargs) 2025-11-03T16:35:43.1927587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1927678Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1927899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1928010Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1928236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1928310Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1928314Z 2025-11-03T16:35:43.1928413Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1928595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1928654Z res = mod(**inputs) 2025-11-03T16:35:43.1928882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1928949Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1929175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1929258Z layer_outputs = layer_module( 2025-11-03T16:35:43.1929465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1929547Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1929766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1929838Z return func(*args, **kwargs) 2025-11-03T16:35:43.1930060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1930150Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1930395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1930502Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1930734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1930816Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1930819Z 2025-11-03T16:35:43.1930923Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1931107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1931166Z res = mod(**inputs) 2025-11-03T16:35:43.1931400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1931465Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1931696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1931761Z layer_outputs = layer_module( 2025-11-03T16:35:43.1931975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1932065Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1932289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1932362Z return func(*args, **kwargs) 2025-11-03T16:35:43.1932581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1932670Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1932905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1933014Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1933241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1933316Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1933321Z 2025-11-03T16:35:43.1933428Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1933609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1933677Z res = mod(**inputs) 2025-11-03T16:35:43.1933906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1933973Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1934203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1934269Z layer_outputs = layer_module( 2025-11-03T16:35:43.1934482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1934554Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1934781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1934871Z return func(*args, **kwargs) 2025-11-03T16:35:43.1935091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1935173Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1935396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1935460Z return func(*args, **kwargs) 2025-11-03T16:35:43.1935687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1935780Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1936010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1936073Z return func(*args, **kwargs) 2025-11-03T16:35:43.1936304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1936378Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1936382Z 2025-11-03T16:35:43.1936476Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1936667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1936726Z res = mod(**inputs) 2025-11-03T16:35:43.1936958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1937025Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1937250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1937323Z layer_outputs = layer_module( 2025-11-03T16:35:43.1937528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1937622Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1937847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1937909Z return func(*args, **kwargs) 2025-11-03T16:35:43.1938137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1938226Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1938458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1938522Z return func(*args, **kwargs) 2025-11-03T16:35:43.1938751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1938828Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1939052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1939124Z return func(*args, **kwargs) 2025-11-03T16:35:43.1939346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1939424Z key_states = self.k(current_states) 2025-11-03T16:35:43.1939427Z 2025-11-03T16:35:43.1939522Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1939707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1939773Z res = mod(**inputs) 2025-11-03T16:35:43.1939998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1940071Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1940294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1940377Z layer_outputs = layer_module( 2025-11-03T16:35:43.1940590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1940664Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1940892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1940955Z return func(*args, **kwargs) 2025-11-03T16:35:43.1941181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1941304Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1941525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1941593Z return func(*args, **kwargs) 2025-11-03T16:35:43.1941835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1941920Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1942143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1942206Z return func(*args, **kwargs) 2025-11-03T16:35:43.1942433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1942505Z value_states = self.v(current_states) 2025-11-03T16:35:43.1942508Z 2025-11-03T16:35:43.1942591Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1942667Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1942761Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1942952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1943013Z res = mod(**inputs) 2025-11-03T16:35:43.1943258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1943326Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1943556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1943621Z layer_outputs = layer_module( 2025-11-03T16:35:43.1943846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1943928Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1944150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1944222Z return func(*args, **kwargs) 2025-11-03T16:35:43.1944442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1944520Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1944748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1944809Z return func(*args, **kwargs) 2025-11-03T16:35:43.1945037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1945113Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1945337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1945409Z return func(*args, **kwargs) 2025-11-03T16:35:43.1945629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1945708Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1945726Z 2025-11-03T16:35:43.1945823Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1946014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1946073Z res = mod(**inputs) 2025-11-03T16:35:43.1946298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1946373Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1946608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1946684Z layer_outputs = layer_module( 2025-11-03T16:35:43.1946913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1946987Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1947218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1947284Z return func(*args, **kwargs) 2025-11-03T16:35:43.1947511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1947586Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1947807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1947878Z return func(*args, **kwargs) 2025-11-03T16:35:43.1948101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1948189Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1948409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1948479Z return func(*args, **kwargs) 2025-11-03T16:35:43.1948715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1948787Z key_states = self.k(current_states) 2025-11-03T16:35:43.1948791Z 2025-11-03T16:35:43.1948894Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1949078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1949145Z res = mod(**inputs) 2025-11-03T16:35:43.1949390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1949459Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1949689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1949755Z layer_outputs = layer_module( 2025-11-03T16:35:43.1949965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1950039Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1950259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1950331Z return func(*args, **kwargs) 2025-11-03T16:35:43.1950551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1950631Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1950855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1950926Z return func(*args, **kwargs) 2025-11-03T16:35:43.1951145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1951220Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1951479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1951542Z return func(*args, **kwargs) 2025-11-03T16:35:43.1951771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1951843Z value_states = self.v(current_states) 2025-11-03T16:35:43.1951846Z 2025-11-03T16:35:43.1951921Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1952003Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1952099Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1952290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1952367Z res = mod(**inputs) 2025-11-03T16:35:43.1952594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1952670Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1952898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1952971Z layer_outputs = layer_module( 2025-11-03T16:35:43.1953179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1953259Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1953482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1953547Z return func(*args, **kwargs) 2025-11-03T16:35:43.1953778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1953854Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1954150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1954220Z return func(*args, **kwargs) 2025-11-03T16:35:43.1954463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1954550Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1954794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1954865Z return func(*args, **kwargs) 2025-11-03T16:35:43.1955109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1955183Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1955195Z 2025-11-03T16:35:43.1955294Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1955484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1955557Z res = mod(**inputs) 2025-11-03T16:35:43.1955792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1955870Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1956099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1956167Z layer_outputs = layer_module( 2025-11-03T16:35:43.1956389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1956464Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1956705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1956769Z return func(*args, **kwargs) 2025-11-03T16:35:43.1957055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.1957159Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.1957390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1957461Z return func(*args, **kwargs) 2025-11-03T16:35:43.1957692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.1957775Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.1958013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1958077Z return func(*args, **kwargs) 2025-11-03T16:35:43.1958332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1958406Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1958412Z 2025-11-03T16:35:43.1958517Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1958706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1958765Z res = mod(**inputs) 2025-11-03T16:35:43.1959003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1959072Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1959312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1959379Z layer_outputs = layer_module( 2025-11-03T16:35:43.1959592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1959677Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1959907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1959980Z return func(*args, **kwargs) 2025-11-03T16:35:43.1960227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1960306Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1960548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1960613Z return func(*args, **kwargs) 2025-11-03T16:35:43.1960866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1960945Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1961185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1961248Z return func(*args, **kwargs) 2025-11-03T16:35:43.1961475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1961560Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1961564Z 2025-11-03T16:35:43.1961662Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1961856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1961915Z res = mod(**inputs) 2025-11-03T16:35:43.1962146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1962222Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1962451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1962527Z layer_outputs = layer_module( 2025-11-03T16:35:43.1962739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1962830Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1963070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1963134Z return func(*args, **kwargs) 2025-11-03T16:35:43.1963367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1963444Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1963678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1963743Z return func(*args, **kwargs) 2025-11-03T16:35:43.1963988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-11-03T16:35:43.1964123Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.1964128Z 2025-11-03T16:35:43.1964226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1964419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1964481Z res = mod(**inputs) 2025-11-03T16:35:43.1964714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1964791Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1965025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1965100Z layer_outputs = layer_module( 2025-11-03T16:35:43.1965313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1965396Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1965629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1965695Z return func(*args, **kwargs) 2025-11-03T16:35:43.1965945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1966034Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1966271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1966400Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1966628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1966732Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1966735Z 2025-11-03T16:35:43.1966833Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1967024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1967086Z res = mod(**inputs) 2025-11-03T16:35:43.1967316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1967391Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1967617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1967691Z layer_outputs = layer_module( 2025-11-03T16:35:43.1967903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1967984Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1968221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1968286Z return func(*args, **kwargs) 2025-11-03T16:35:43.1968517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1968623Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1968855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1968965Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1969193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1969276Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1969279Z 2025-11-03T16:35:43.1969377Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1969587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1969647Z res = mod(**inputs) 2025-11-03T16:35:43.1969884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1969963Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1970191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1970263Z layer_outputs = layer_module( 2025-11-03T16:35:43.1970467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1970545Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1970767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1970831Z return func(*args, **kwargs) 2025-11-03T16:35:43.1971057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1971141Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1971381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1971491Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1971709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1971798Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1971802Z 2025-11-03T16:35:43.1971897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1972112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1972174Z res = mod(**inputs) 2025-11-03T16:35:43.1972407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1749, in forward 2025-11-03T16:35:43.1972474Z encoder_outputs = self.encoder( 2025-11-03T16:35:43.1972698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1972775Z layer_outputs = layer_module( 2025-11-03T16:35:43.1972981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1973060Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1973284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1973350Z return func(*args, **kwargs) 2025-11-03T16:35:43.1973577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1973661Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1973894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1974002Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1974251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1974327Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1974331Z 2025-11-03T16:35:43.1974428Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1974625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1974686Z res = mod(**inputs) 2025-11-03T16:35:43.1974925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1975011Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1975244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1975319Z layer_outputs = layer_module( 2025-11-03T16:35:43.1975533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1975618Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1975850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1975925Z return func(*args, **kwargs) 2025-11-03T16:35:43.1976153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.1976230Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.1976460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1976525Z return func(*args, **kwargs) 2025-11-03T16:35:43.1976760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.1976842Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.1977082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1977154Z return func(*args, **kwargs) 2025-11-03T16:35:43.1977377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.1977454Z key_states = self.k(current_states) 2025-11-03T16:35:43.1977458Z 2025-11-03T16:35:43.1977586Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1977771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1977839Z res = mod(**inputs) 2025-11-03T16:35:43.1978066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1978139Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1978362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1978429Z layer_outputs = layer_module( 2025-11-03T16:35:43.1978641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1978713Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1978942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1979006Z return func(*args, **kwargs) 2025-11-03T16:35:43.1979234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.1979310Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.1979531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1979617Z return func(*args, **kwargs) 2025-11-03T16:35:43.1979839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.1979927Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.1980152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1980215Z return func(*args, **kwargs) 2025-11-03T16:35:43.1980447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.1980520Z value_states = self.v(current_states) 2025-11-03T16:35:43.1980539Z 2025-11-03T16:35:43.1980622Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1980697Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.1980793Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1980985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1981047Z res = mod(**inputs) 2025-11-03T16:35:43.1981294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1981360Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1981593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1981659Z layer_outputs = layer_module( 2025-11-03T16:35:43.1981866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1981947Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1982171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1982243Z return func(*args, **kwargs) 2025-11-03T16:35:43.1982472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.1982570Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.1982809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1982877Z return func(*args, **kwargs) 2025-11-03T16:35:43.1983120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.1983219Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.1983449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1983523Z return func(*args, **kwargs) 2025-11-03T16:35:43.1983750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.1983833Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.1983837Z 2025-11-03T16:35:43.1983935Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1984128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1984188Z res = mod(**inputs) 2025-11-03T16:35:43.1984422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1984499Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1984722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1984795Z layer_outputs = layer_module( 2025-11-03T16:35:43.1985002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1985074Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1985307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1985386Z return func(*args, **kwargs) 2025-11-03T16:35:43.1985617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1985701Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1985924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1986042Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1986267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.1986384Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.1986387Z 2025-11-03T16:35:43.1986483Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1986681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1986742Z res = mod(**inputs) 2025-11-03T16:35:43.1986979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1987054Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1987283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1987358Z layer_outputs = layer_module( 2025-11-03T16:35:43.1987567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1987640Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1987875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1987938Z return func(*args, **kwargs) 2025-11-03T16:35:43.1988184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1988271Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1988498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1988605Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1988836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.1988918Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.1988921Z 2025-11-03T16:35:43.1989017Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1989208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1989267Z res = mod(**inputs) 2025-11-03T16:35:43.1989507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1989584Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1989813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1989886Z layer_outputs = layer_module( 2025-11-03T16:35:43.1990098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1990170Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1990406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1990473Z return func(*args, **kwargs) 2025-11-03T16:35:43.1990703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1990802Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1991035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1991140Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1991359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.1991448Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.1991451Z 2025-11-03T16:35:43.1991547Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1991736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1991812Z res = mod(**inputs) 2025-11-03T16:35:43.1992039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1992112Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1992340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1992413Z layer_outputs = layer_module( 2025-11-03T16:35:43.1992621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1992701Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1992926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1992988Z return func(*args, **kwargs) 2025-11-03T16:35:43.1993219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.1993305Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.1993536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.1993642Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.1993894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.1993978Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.1993982Z 2025-11-03T16:35:43.1994150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1994350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1994442Z res = mod(**inputs) 2025-11-03T16:35:43.1994680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1994760Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1995001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1995078Z layer_outputs = layer_module( 2025-11-03T16:35:43.1995290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1995374Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1995604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1995669Z return func(*args, **kwargs) 2025-11-03T16:35:43.1995902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1995980Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1996215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1996280Z return func(*args, **kwargs) 2025-11-03T16:35:43.1996504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1996611Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1996845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1996916Z return func(*args, **kwargs) 2025-11-03T16:35:43.1997145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.1997220Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.1997230Z 2025-11-03T16:35:43.1997329Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.1997515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.1997601Z res = mod(**inputs) 2025-11-03T16:35:43.1997838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.1997914Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.1998145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.1998211Z layer_outputs = layer_module( 2025-11-03T16:35:43.1998433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.1998507Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.1998744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1998808Z return func(*args, **kwargs) 2025-11-03T16:35:43.1999036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.1999123Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.1999354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.1999428Z return func(*args, **kwargs) 2025-11-03T16:35:43.1999671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.1999752Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.1999989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2000053Z return func(*args, **kwargs) 2025-11-03T16:35:43.2000310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2000385Z key_states = self.k(current_states) 2025-11-03T16:35:43.2000390Z 2025-11-03T16:35:43.2000495Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2000688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2000750Z res = mod(**inputs) 2025-11-03T16:35:43.2000987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2001056Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2001290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2001357Z layer_outputs = layer_module( 2025-11-03T16:35:43.2001570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2001650Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2001880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2001953Z return func(*args, **kwargs) 2025-11-03T16:35:43.2002181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2002282Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2002514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2002578Z return func(*args, **kwargs) 2025-11-03T16:35:43.2002814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2002892Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2003133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2003196Z return func(*args, **kwargs) 2025-11-03T16:35:43.2003450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2003534Z value_states = self.v(current_states) 2025-11-03T16:35:43.2003538Z 2025-11-03T16:35:43.2003618Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2003702Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2003801Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2003987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2004055Z res = mod(**inputs) 2025-11-03T16:35:43.2004288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2004365Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2004601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2004669Z layer_outputs = layer_module( 2025-11-03T16:35:43.2004890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2004964Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2005229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2005294Z return func(*args, **kwargs) 2025-11-03T16:35:43.2005531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2005605Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2005853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2005927Z return func(*args, **kwargs) 2025-11-03T16:35:43.2006160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2006247Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2006479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2006544Z return func(*args, **kwargs) 2025-11-03T16:35:43.2006790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2006862Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2006865Z 2025-11-03T16:35:43.2006970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2007168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2007236Z res = mod(**inputs) 2025-11-03T16:35:43.2007464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2007532Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2007764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2007828Z layer_outputs = layer_module( 2025-11-03T16:35:43.2008059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2008133Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2008355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2008426Z return func(*args, **kwargs) 2025-11-03T16:35:43.2008648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2008733Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2008956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2009033Z return func(*args, **kwargs) 2025-11-03T16:35:43.2009267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2009348Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2009581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2009644Z return func(*args, **kwargs) 2025-11-03T16:35:43.2009875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2009948Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2009951Z 2025-11-03T16:35:43.2010047Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2010237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2010297Z res = mod(**inputs) 2025-11-03T16:35:43.2010531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2010599Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2010839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2010914Z layer_outputs = layer_module( 2025-11-03T16:35:43.2011124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2011204Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2011445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2011510Z return func(*args, **kwargs) 2025-11-03T16:35:43.2011739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2011814Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2012042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2012107Z return func(*args, **kwargs) 2025-11-03T16:35:43.2012336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2012415Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2012639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2012708Z return func(*args, **kwargs) 2025-11-03T16:35:43.2012932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2013011Z key_states = self.k(current_states) 2025-11-03T16:35:43.2013015Z 2025-11-03T16:35:43.2013111Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2013436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2013511Z res = mod(**inputs) 2025-11-03T16:35:43.2013777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2013854Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2014080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2014148Z layer_outputs = layer_module( 2025-11-03T16:35:43.2014365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2014440Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2014672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2014769Z return func(*args, **kwargs) 2025-11-03T16:35:43.2014999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2015076Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2015303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2015376Z return func(*args, **kwargs) 2025-11-03T16:35:43.2015599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2015685Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2015915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2015979Z return func(*args, **kwargs) 2025-11-03T16:35:43.2016213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2016286Z value_states = self.v(current_states) 2025-11-03T16:35:43.2016290Z 2025-11-03T16:35:43.2016374Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2016449Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2016576Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2016769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2016829Z res = mod(**inputs) 2025-11-03T16:35:43.2017063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2017132Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2017384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2017454Z layer_outputs = layer_module( 2025-11-03T16:35:43.2017659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2017740Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2017964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2018033Z return func(*args, **kwargs) 2025-11-03T16:35:43.2018254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2018326Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2018558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2018619Z return func(*args, **kwargs) 2025-11-03T16:35:43.2018848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2018929Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2019153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2019238Z return func(*args, **kwargs) 2025-11-03T16:35:43.2019463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2019541Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2019545Z 2025-11-03T16:35:43.2019640Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2019834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2019895Z res = mod(**inputs) 2025-11-03T16:35:43.2020124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2020213Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2020439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2020512Z layer_outputs = layer_module( 2025-11-03T16:35:43.2020719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2020793Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2021024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2021086Z return func(*args, **kwargs) 2025-11-03T16:35:43.2021320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2021406Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2021628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2021746Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2021970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2022070Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2022087Z 2025-11-03T16:35:43.2022183Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2022373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2022432Z res = mod(**inputs) 2025-11-03T16:35:43.2022655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2022747Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2022970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2023045Z layer_outputs = layer_module( 2025-11-03T16:35:43.2023249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2023322Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2023556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2023621Z return func(*args, **kwargs) 2025-11-03T16:35:43.2023848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2023931Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2024153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2024267Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2024490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2024572Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2024575Z 2025-11-03T16:35:43.2024669Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2024876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2024934Z res = mod(**inputs) 2025-11-03T16:35:43.2025161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2025237Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2025465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2025538Z layer_outputs = layer_module( 2025-11-03T16:35:43.2025744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2025831Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2026065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2026131Z return func(*args, **kwargs) 2025-11-03T16:35:43.2026362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2026444Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2026676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2026782Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2027013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2027104Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2027109Z 2025-11-03T16:35:43.2027205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2027398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2027460Z res = mod(**inputs) 2025-11-03T16:35:43.2027701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2027778Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2027998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2028071Z layer_outputs = layer_module( 2025-11-03T16:35:43.2028287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2028360Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2028589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2028653Z return func(*args, **kwargs) 2025-11-03T16:35:43.2028880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2028965Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2029194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2029300Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2029520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2029601Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2029606Z 2025-11-03T16:35:43.2029702Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2029891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2029951Z res = mod(**inputs) 2025-11-03T16:35:43.2030180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2030294Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2030522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2030593Z layer_outputs = layer_module( 2025-11-03T16:35:43.2030802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2030883Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2031110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2031174Z return func(*args, **kwargs) 2025-11-03T16:35:43.2031424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2031498Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2031726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2031793Z return func(*args, **kwargs) 2025-11-03T16:35:43.2032015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2032101Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2032325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2032395Z return func(*args, **kwargs) 2025-11-03T16:35:43.2032619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2032693Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2032703Z 2025-11-03T16:35:43.2032798Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2032979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2033046Z res = mod(**inputs) 2025-11-03T16:35:43.2033297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2033375Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2033604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2033669Z layer_outputs = layer_module( 2025-11-03T16:35:43.2033904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2033980Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2034274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2034342Z return func(*args, **kwargs) 2025-11-03T16:35:43.2034571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2034659Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2034923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2034991Z return func(*args, **kwargs) 2025-11-03T16:35:43.2035211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2035288Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2035518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2035583Z return func(*args, **kwargs) 2025-11-03T16:35:43.2035811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2035884Z key_states = self.k(current_states) 2025-11-03T16:35:43.2035913Z 2025-11-03T16:35:43.2036017Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2036203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2036263Z res = mod(**inputs) 2025-11-03T16:35:43.2036495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2036564Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2036795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2036861Z layer_outputs = layer_module( 2025-11-03T16:35:43.2037072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2037169Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2037394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2037466Z return func(*args, **kwargs) 2025-11-03T16:35:43.2037690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2037764Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2037997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2038059Z return func(*args, **kwargs) 2025-11-03T16:35:43.2038292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2038369Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2038603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2038666Z return func(*args, **kwargs) 2025-11-03T16:35:43.2038889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2038983Z value_states = self.v(current_states) 2025-11-03T16:35:43.2038987Z 2025-11-03T16:35:43.2039064Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2039145Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2039243Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2039426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2039508Z res = mod(**inputs) 2025-11-03T16:35:43.2039736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2039816Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2040041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2040107Z layer_outputs = layer_module( 2025-11-03T16:35:43.2040324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2040398Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2040630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2040694Z return func(*args, **kwargs) 2025-11-03T16:35:43.2040924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2041000Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2041221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2041295Z return func(*args, **kwargs) 2025-11-03T16:35:43.2041517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2041615Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2041838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2041899Z return func(*args, **kwargs) 2025-11-03T16:35:43.2042128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2042198Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2042202Z 2025-11-03T16:35:43.2042303Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2042485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2042561Z res = mod(**inputs) 2025-11-03T16:35:43.2042794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2042866Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2043098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2043165Z layer_outputs = layer_module( 2025-11-03T16:35:43.2043380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2043454Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2043678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2043748Z return func(*args, **kwargs) 2025-11-03T16:35:43.2043968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2044050Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2044273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2044341Z return func(*args, **kwargs) 2025-11-03T16:35:43.2044587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-11-03T16:35:43.2044715Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.2044718Z 2025-11-03T16:35:43.2044822Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2045021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2045091Z res = mod(**inputs) 2025-11-03T16:35:43.2045315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2045384Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2045614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2045682Z layer_outputs = layer_module( 2025-11-03T16:35:43.2045897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2045971Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2046195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2046265Z return func(*args, **kwargs) 2025-11-03T16:35:43.2046488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2046570Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2046792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2046856Z return func(*args, **kwargs) 2025-11-03T16:35:43.2047083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2047177Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2047406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2047468Z return func(*args, **kwargs) 2025-11-03T16:35:43.2047700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2047772Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2047777Z 2025-11-03T16:35:43.2047871Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2048057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2048134Z res = mod(**inputs) 2025-11-03T16:35:43.2048364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2048432Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2048655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2048728Z layer_outputs = layer_module( 2025-11-03T16:35:43.2048936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2049013Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2049242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2049304Z return func(*args, **kwargs) 2025-11-03T16:35:43.2049533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2049608Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2049837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2049901Z return func(*args, **kwargs) 2025-11-03T16:35:43.2050145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2050223Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2050444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2050515Z return func(*args, **kwargs) 2025-11-03T16:35:43.2050751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2050830Z key_states = self.k(current_states) 2025-11-03T16:35:43.2050833Z 2025-11-03T16:35:43.2050926Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2051109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2051178Z res = mod(**inputs) 2025-11-03T16:35:43.2051407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2051479Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2051707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2051771Z layer_outputs = layer_module( 2025-11-03T16:35:43.2051991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2052063Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2052298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2052363Z return func(*args, **kwargs) 2025-11-03T16:35:43.2052598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2052689Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2052912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2052983Z return func(*args, **kwargs) 2025-11-03T16:35:43.2053203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2053290Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2053515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2053593Z return func(*args, **kwargs) 2025-11-03T16:35:43.2053828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2053902Z value_states = self.v(current_states) 2025-11-03T16:35:43.2053907Z 2025-11-03T16:35:43.2053990Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2054067Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2054162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2054351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2054411Z res = mod(**inputs) 2025-11-03T16:35:43.2054646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2054714Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2054946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2055013Z layer_outputs = layer_module( 2025-11-03T16:35:43.2055219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2055300Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2055539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2055611Z return func(*args, **kwargs) 2025-11-03T16:35:43.2055843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2055917Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2056171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2056236Z return func(*args, **kwargs) 2025-11-03T16:35:43.2056465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2056544Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2056769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2056840Z return func(*args, **kwargs) 2025-11-03T16:35:43.2057067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2057146Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2057150Z 2025-11-03T16:35:43.2057245Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2057438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2057499Z res = mod(**inputs) 2025-11-03T16:35:43.2057725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2057802Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2058029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2058126Z layer_outputs = layer_module( 2025-11-03T16:35:43.2058335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2058407Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2058641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2058703Z return func(*args, **kwargs) 2025-11-03T16:35:43.2058934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2059016Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2059239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2059371Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2059589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2059691Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2059694Z 2025-11-03T16:35:43.2059790Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2059980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2060038Z res = mod(**inputs) 2025-11-03T16:35:43.2060260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2060335Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2060556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2060630Z layer_outputs = layer_module( 2025-11-03T16:35:43.2060835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2060908Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2061152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2061217Z return func(*args, **kwargs) 2025-11-03T16:35:43.2061449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2061533Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2061778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2061889Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2062112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2062191Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2062195Z 2025-11-03T16:35:43.2062293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2062484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2062544Z res = mod(**inputs) 2025-11-03T16:35:43.2062777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2062853Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2063078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2063151Z layer_outputs = layer_module( 2025-11-03T16:35:43.2063357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2063429Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2063660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2063739Z return func(*args, **kwargs) 2025-11-03T16:35:43.2063973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2064057Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2064286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2064393Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2064615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2064721Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2064724Z 2025-11-03T16:35:43.2064820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2065012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2065073Z res = mod(**inputs) 2025-11-03T16:35:43.2065313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2065387Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2065611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2065684Z layer_outputs = layer_module( 2025-11-03T16:35:43.2065892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2065972Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2066199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2066263Z return func(*args, **kwargs) 2025-11-03T16:35:43.2066496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2066594Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2066836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2066944Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2067163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2067260Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2067264Z 2025-11-03T16:35:43.2067362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2067551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2067612Z res = mod(**inputs) 2025-11-03T16:35:43.2067840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2067918Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2068146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2068222Z layer_outputs = layer_module( 2025-11-03T16:35:43.2068431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2068514Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2068742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2068808Z return func(*args, **kwargs) 2025-11-03T16:35:43.2069041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2069121Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2069354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2069435Z return func(*args, **kwargs) 2025-11-03T16:35:43.2069657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2069742Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2069965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2070038Z return func(*args, **kwargs) 2025-11-03T16:35:43.2070257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2070344Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2070354Z 2025-11-03T16:35:43.2070450Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2070634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2070705Z res = mod(**inputs) 2025-11-03T16:35:43.2070929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2071003Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2071230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2071294Z layer_outputs = layer_module( 2025-11-03T16:35:43.2071510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2071585Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2071828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2071892Z return func(*args, **kwargs) 2025-11-03T16:35:43.2072120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2072221Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2072449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2072520Z return func(*args, **kwargs) 2025-11-03T16:35:43.2072746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2072847Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2073079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2073146Z return func(*args, **kwargs) 2025-11-03T16:35:43.2073381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2073454Z key_states = self.k(current_states) 2025-11-03T16:35:43.2073459Z 2025-11-03T16:35:43.2073564Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2073752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2073813Z res = mod(**inputs) 2025-11-03T16:35:43.2074175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2074253Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2074502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2074572Z layer_outputs = layer_module( 2025-11-03T16:35:43.2074794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2074880Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2075121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2075217Z return func(*args, **kwargs) 2025-11-03T16:35:43.2075461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2075559Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2075795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2075861Z return func(*args, **kwargs) 2025-11-03T16:35:43.2076101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2076194Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2076435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2076500Z return func(*args, **kwargs) 2025-11-03T16:35:43.2076730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2076813Z value_states = self.v(current_states) 2025-11-03T16:35:43.2076817Z 2025-11-03T16:35:43.2076894Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2076978Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2077077Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2077266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2077338Z res = mod(**inputs) 2025-11-03T16:35:43.2077568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2077644Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2077873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2077941Z layer_outputs = layer_module( 2025-11-03T16:35:43.2078182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2078258Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2078494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2078558Z return func(*args, **kwargs) 2025-11-03T16:35:43.2078813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2078892Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2079122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2079196Z return func(*args, **kwargs) 2025-11-03T16:35:43.2079424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2079511Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2079741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2079806Z return func(*args, **kwargs) 2025-11-03T16:35:43.2080042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2080116Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2080119Z 2025-11-03T16:35:43.2080226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2080414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2080485Z res = mod(**inputs) 2025-11-03T16:35:43.2080719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2080808Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2081055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2081123Z layer_outputs = layer_module( 2025-11-03T16:35:43.2081345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2081420Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2081657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2081728Z return func(*args, **kwargs) 2025-11-03T16:35:43.2081969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2082054Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2082288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2082356Z return func(*args, **kwargs) 2025-11-03T16:35:43.2082589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2082668Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2082905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2082972Z return func(*args, **kwargs) 2025-11-03T16:35:43.2083207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2083283Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2083287Z 2025-11-03T16:35:43.2083383Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2083578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2083641Z res = mod(**inputs) 2025-11-03T16:35:43.2083895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2083967Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2084202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2084278Z layer_outputs = layer_module( 2025-11-03T16:35:43.2084510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2084592Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2084819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2084884Z return func(*args, **kwargs) 2025-11-03T16:35:43.2085118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2085197Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2085432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2085499Z return func(*args, **kwargs) 2025-11-03T16:35:43.2085731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2085813Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2086040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2086113Z return func(*args, **kwargs) 2025-11-03T16:35:43.2086341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2086421Z key_states = self.k(current_states) 2025-11-03T16:35:43.2086442Z 2025-11-03T16:35:43.2086542Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2086732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2086802Z res = mod(**inputs) 2025-11-03T16:35:43.2087031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2087107Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2087337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2087403Z layer_outputs = layer_module( 2025-11-03T16:35:43.2087634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2087706Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2087945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2088011Z return func(*args, **kwargs) 2025-11-03T16:35:43.2088243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2088319Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2088547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2088619Z return func(*args, **kwargs) 2025-11-03T16:35:43.2088849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2088936Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2089163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2089226Z return func(*args, **kwargs) 2025-11-03T16:35:43.2089479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2089553Z value_states = self.v(current_states) 2025-11-03T16:35:43.2089557Z 2025-11-03T16:35:43.2089640Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2089716Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2089812Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2090021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2090083Z res = mod(**inputs) 2025-11-03T16:35:43.2090320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2090392Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2090639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2090706Z layer_outputs = layer_module( 2025-11-03T16:35:43.2090913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2090995Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2091219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2091291Z return func(*args, **kwargs) 2025-11-03T16:35:43.2091512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2091586Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2091815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2091878Z return func(*args, **kwargs) 2025-11-03T16:35:43.2092108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2092201Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2092424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2092495Z return func(*args, **kwargs) 2025-11-03T16:35:43.2092715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2092793Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2092798Z 2025-11-03T16:35:43.2092893Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2093080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2093155Z res = mod(**inputs) 2025-11-03T16:35:43.2093381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2093459Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2093684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2093757Z layer_outputs = layer_module( 2025-11-03T16:35:43.2093962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2094034Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2094265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2094328Z return func(*args, **kwargs) 2025-11-03T16:35:43.2094555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2094630Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2094851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2094923Z return func(*args, **kwargs) 2025-11-03T16:35:43.2095165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 528, in forward 2025-11-03T16:35:43.2095295Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.2095298Z 2025-11-03T16:35:43.2095394Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2095606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2095667Z res = mod(**inputs) 2025-11-03T16:35:43.2095891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2095969Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2096190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2096262Z layer_outputs = layer_module( 2025-11-03T16:35:43.2096471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2096542Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2096770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2096833Z return func(*args, **kwargs) 2025-11-03T16:35:43.2097060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2097144Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2097366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2097481Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2097717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2097818Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2097822Z 2025-11-03T16:35:43.2097915Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2098107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2098167Z res = mod(**inputs) 2025-11-03T16:35:43.2098390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2098466Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2098710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2098782Z layer_outputs = layer_module( 2025-11-03T16:35:43.2098987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2099061Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2099290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2099353Z return func(*args, **kwargs) 2025-11-03T16:35:43.2099579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2099662Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2099887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2099995Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2100213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2100297Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2100302Z 2025-11-03T16:35:43.2100395Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2100598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2100658Z res = mod(**inputs) 2025-11-03T16:35:43.2100884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2100959Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2101196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2101270Z layer_outputs = layer_module( 2025-11-03T16:35:43.2101478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2101549Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2101780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2101846Z return func(*args, **kwargs) 2025-11-03T16:35:43.2102075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2102159Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2102385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2102493Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2102716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2102805Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2102808Z 2025-11-03T16:35:43.2102906Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2103095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2103169Z res = mod(**inputs) 2025-11-03T16:35:43.2103400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2103474Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2103697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2103769Z layer_outputs = layer_module( 2025-11-03T16:35:43.2103979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2104061Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2104299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2104363Z return func(*args, **kwargs) 2025-11-03T16:35:43.2104590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2104674Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2104900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2105006Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2105228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2105313Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2105316Z 2025-11-03T16:35:43.2105410Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2105602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2105662Z res = mod(**inputs) 2025-11-03T16:35:43.2105884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2105974Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2106198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2106273Z layer_outputs = layer_module( 2025-11-03T16:35:43.2106479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2106574Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2106800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2106866Z return func(*args, **kwargs) 2025-11-03T16:35:43.2107102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2107178Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2107414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2107479Z return func(*args, **kwargs) 2025-11-03T16:35:43.2107701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2107785Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2108010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2108080Z return func(*args, **kwargs) 2025-11-03T16:35:43.2108301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2108382Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2108385Z 2025-11-03T16:35:43.2108479Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2108676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2108745Z res = mod(**inputs) 2025-11-03T16:35:43.2108969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2109043Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2109265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2109330Z layer_outputs = layer_module( 2025-11-03T16:35:43.2109544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2109638Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2109866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2109929Z return func(*args, **kwargs) 2025-11-03T16:35:43.2110151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2110232Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2110455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2110524Z return func(*args, **kwargs) 2025-11-03T16:35:43.2110747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2110830Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2111050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2111115Z return func(*args, **kwargs) 2025-11-03T16:35:43.2111342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2111414Z key_states = self.k(current_states) 2025-11-03T16:35:43.2111417Z 2025-11-03T16:35:43.2111535Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2111718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2111778Z res = mod(**inputs) 2025-11-03T16:35:43.2112011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2112093Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2112327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2112394Z layer_outputs = layer_module( 2025-11-03T16:35:43.2112602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2112683Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2112934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2113006Z return func(*args, **kwargs) 2025-11-03T16:35:43.2113425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2113519Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2113754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2113821Z return func(*args, **kwargs) 2025-11-03T16:35:43.2114097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2114184Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2114428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2114533Z return func(*args, **kwargs) 2025-11-03T16:35:43.2114777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2114865Z value_states = self.v(current_states) 2025-11-03T16:35:43.2114869Z 2025-11-03T16:35:43.2114949Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2115033Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2115134Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2115343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2115409Z res = mod(**inputs) 2025-11-03T16:35:43.2115663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2115739Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2115969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2116047Z layer_outputs = layer_module( 2025-11-03T16:35:43.2116258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2116333Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2116621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2116687Z return func(*args, **kwargs) 2025-11-03T16:35:43.2116926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2117003Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2117238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2117313Z return func(*args, **kwargs) 2025-11-03T16:35:43.2117564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2117651Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2117883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2117947Z return func(*args, **kwargs) 2025-11-03T16:35:43.2118189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2118316Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2118321Z 2025-11-03T16:35:43.2118428Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2118617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2118689Z res = mod(**inputs) 2025-11-03T16:35:43.2118916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2118988Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2119224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2119292Z layer_outputs = layer_module( 2025-11-03T16:35:43.2119511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2119585Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2119817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2119892Z return func(*args, **kwargs) 2025-11-03T16:35:43.2120120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2120205Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2120466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2120530Z return func(*args, **kwargs) 2025-11-03T16:35:43.2120767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2120849Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2121087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2121153Z return func(*args, **kwargs) 2025-11-03T16:35:43.2121395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2121487Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2121490Z 2025-11-03T16:35:43.2121589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2121788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2121851Z res = mod(**inputs) 2025-11-03T16:35:43.2122090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2122158Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2122389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2122463Z layer_outputs = layer_module( 2025-11-03T16:35:43.2122675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2122755Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2122982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2123047Z return func(*args, **kwargs) 2025-11-03T16:35:43.2123281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2123369Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2123604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2123667Z return func(*args, **kwargs) 2025-11-03T16:35:43.2123903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2123997Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2124223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2124297Z return func(*args, **kwargs) 2025-11-03T16:35:43.2124534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2124618Z key_states = self.k(current_states) 2025-11-03T16:35:43.2124621Z 2025-11-03T16:35:43.2124721Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2124912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2124980Z res = mod(**inputs) 2025-11-03T16:35:43.2125212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2125288Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2125521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2125590Z layer_outputs = layer_module( 2025-11-03T16:35:43.2125807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2125881Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2126132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2126199Z return func(*args, **kwargs) 2025-11-03T16:35:43.2126437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2126515Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2126742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2126817Z return func(*args, **kwargs) 2025-11-03T16:35:43.2127056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2127153Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2127377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2127444Z return func(*args, **kwargs) 2025-11-03T16:35:43.2127675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2127748Z value_states = self.v(current_states) 2025-11-03T16:35:43.2127751Z 2025-11-03T16:35:43.2127831Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2127903Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2127998Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2128189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2128249Z res = mod(**inputs) 2025-11-03T16:35:43.2128481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2128549Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2128782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2128849Z layer_outputs = layer_module( 2025-11-03T16:35:43.2129071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2129151Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2129376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2129446Z return func(*args, **kwargs) 2025-11-03T16:35:43.2129686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2129760Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2129989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2130053Z return func(*args, **kwargs) 2025-11-03T16:35:43.2130288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2130370Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2130594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2130665Z return func(*args, **kwargs) 2025-11-03T16:35:43.2130887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2130965Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2130968Z 2025-11-03T16:35:43.2131065Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2131256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2131316Z res = mod(**inputs) 2025-11-03T16:35:43.2131543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2131636Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2131869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2131942Z layer_outputs = layer_module( 2025-11-03T16:35:43.2132153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2132225Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2132459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2132523Z return func(*args, **kwargs) 2025-11-03T16:35:43.2132768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2132854Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2133077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2133196Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2133417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2133518Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2133522Z 2025-11-03T16:35:43.2133617Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2133811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2133870Z res = mod(**inputs) 2025-11-03T16:35:43.2134095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2134171Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2134393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2134486Z layer_outputs = layer_module( 2025-11-03T16:35:43.2134696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2134769Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2135006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2135071Z return func(*args, **kwargs) 2025-11-03T16:35:43.2135316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2135401Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2135628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2135734Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2135956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2136036Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2136039Z 2025-11-03T16:35:43.2136135Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2136324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2136383Z res = mod(**inputs) 2025-11-03T16:35:43.2136612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2136685Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2136907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2136979Z layer_outputs = layer_module( 2025-11-03T16:35:43.2137184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2137276Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2137507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2137570Z return func(*args, **kwargs) 2025-11-03T16:35:43.2137798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2137882Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2138109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2138231Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2138453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2138546Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2138549Z 2025-11-03T16:35:43.2138646Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2138836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2138896Z res = mod(**inputs) 2025-11-03T16:35:43.2139119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2139195Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2139418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2139493Z layer_outputs = layer_module( 2025-11-03T16:35:43.2139700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2139780Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2140024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2140088Z return func(*args, **kwargs) 2025-11-03T16:35:43.2140317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2140400Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2140650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2140755Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2140975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2141058Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2141062Z 2025-11-03T16:35:43.2141155Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2141345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2141408Z res = mod(**inputs) 2025-11-03T16:35:43.2141637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2141711Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2141936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2142010Z layer_outputs = layer_module( 2025-11-03T16:35:43.2142216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2142297Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2142519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2142599Z return func(*args, **kwargs) 2025-11-03T16:35:43.2142831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2142913Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2143147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 218, in forward 2025-11-03T16:35:43.2143269Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-11-03T16:35:43.2143272Z 2025-11-03T16:35:43.2143369Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2143559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2143636Z res = mod(**inputs) 2025-11-03T16:35:43.2143868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2143934Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2144168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2144234Z layer_outputs = layer_module( 2025-11-03T16:35:43.2144439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2144520Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2144744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2144813Z return func(*args, **kwargs) 2025-11-03T16:35:43.2145035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2145111Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2145342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2145407Z return func(*args, **kwargs) 2025-11-03T16:35:43.2145650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2145730Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2145954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2146024Z return func(*args, **kwargs) 2025-11-03T16:35:43.2146258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2146338Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2146343Z 2025-11-03T16:35:43.2146440Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2146632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2146692Z res = mod(**inputs) 2025-11-03T16:35:43.2146927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2147001Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2147226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2147300Z layer_outputs = layer_module( 2025-11-03T16:35:43.2147508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2147582Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2147813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2147879Z return func(*args, **kwargs) 2025-11-03T16:35:43.2148108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2148201Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2148424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2148494Z return func(*args, **kwargs) 2025-11-03T16:35:43.2148715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2148799Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2149021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2149091Z return func(*args, **kwargs) 2025-11-03T16:35:43.2149330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2149401Z key_states = self.k(current_states) 2025-11-03T16:35:43.2149404Z 2025-11-03T16:35:43.2149505Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2149694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2149761Z res = mod(**inputs) 2025-11-03T16:35:43.2149985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2150052Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2150289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2150355Z layer_outputs = layer_module( 2025-11-03T16:35:43.2150569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2150643Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2150867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2150939Z return func(*args, **kwargs) 2025-11-03T16:35:43.2151175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2151262Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2151484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2151556Z return func(*args, **kwargs) 2025-11-03T16:35:43.2151790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2151867Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2152095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2152157Z return func(*args, **kwargs) 2025-11-03T16:35:43.2152385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2152459Z value_states = self.v(current_states) 2025-11-03T16:35:43.2152464Z 2025-11-03T16:35:43.2152540Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2152619Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2152713Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2152903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2152963Z res = mod(**inputs) 2025-11-03T16:35:43.2153194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2153272Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2153496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2153569Z layer_outputs = layer_module( 2025-11-03T16:35:43.2153801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2153889Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2154187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2154258Z return func(*args, **kwargs) 2025-11-03T16:35:43.2154498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2154580Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2154827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2154917Z return func(*args, **kwargs) 2025-11-03T16:35:43.2155173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2155261Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2155484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2155555Z return func(*args, **kwargs) 2025-11-03T16:35:43.2155788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2155863Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2155875Z 2025-11-03T16:35:43.2155978Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2156168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2156241Z res = mod(**inputs) 2025-11-03T16:35:43.2156479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2156558Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2156793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2156879Z layer_outputs = layer_module( 2025-11-03T16:35:43.2157107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2157187Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2157473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2157558Z return func(*args, **kwargs) 2025-11-03T16:35:43.2157794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2157883Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2158118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2158195Z return func(*args, **kwargs) 2025-11-03T16:35:43.2158432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2158514Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2158757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2158823Z return func(*args, **kwargs) 2025-11-03T16:35:43.2159066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2159141Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2159145Z 2025-11-03T16:35:43.2159254Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2159449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2159513Z res = mod(**inputs) 2025-11-03T16:35:43.2159762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2159850Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2160094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2160164Z layer_outputs = layer_module( 2025-11-03T16:35:43.2160382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2160466Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2160703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2160799Z return func(*args, **kwargs) 2025-11-03T16:35:43.2161041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2161120Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2161373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2161442Z return func(*args, **kwargs) 2025-11-03T16:35:43.2161686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2161767Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2162017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2162083Z return func(*args, **kwargs) 2025-11-03T16:35:43.2162321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2162403Z key_states = self.k(current_states) 2025-11-03T16:35:43.2162407Z 2025-11-03T16:35:43.2162508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2162715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2162803Z res = mod(**inputs) 2025-11-03T16:35:43.2163039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2163118Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2163355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2163445Z layer_outputs = layer_module( 2025-11-03T16:35:43.2163661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2163744Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2163981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2164046Z return func(*args, **kwargs) 2025-11-03T16:35:43.2164289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2164365Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2164606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2164670Z return func(*args, **kwargs) 2025-11-03T16:35:43.2164908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2164997Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2165232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2165306Z return func(*args, **kwargs) 2025-11-03T16:35:43.2165541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2165634Z value_states = self.v(current_states) 2025-11-03T16:35:43.2165647Z 2025-11-03T16:35:43.2165726Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2165803Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2165920Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2166107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2166167Z res = mod(**inputs) 2025-11-03T16:35:43.2166406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2166474Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2166725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2166792Z layer_outputs = layer_module( 2025-11-03T16:35:43.2167006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2167082Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2167312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2167382Z return func(*args, **kwargs) 2025-11-03T16:35:43.2167609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2167694Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2167923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2167988Z return func(*args, **kwargs) 2025-11-03T16:35:43.2168223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2168303Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2168555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2168620Z return func(*args, **kwargs) 2025-11-03T16:35:43.2168848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2168925Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2168929Z 2025-11-03T16:35:43.2169025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2169235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2169297Z res = mod(**inputs) 2025-11-03T16:35:43.2169533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2169602Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2169837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2169914Z layer_outputs = layer_module( 2025-11-03T16:35:43.2170124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2170205Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2170435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2170499Z return func(*args, **kwargs) 2025-11-03T16:35:43.2170734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2170820Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2171057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2171169Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2171422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2171516Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2171519Z 2025-11-03T16:35:43.2171615Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2171809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2171871Z res = mod(**inputs) 2025-11-03T16:35:43.2172113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2172199Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2172427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2172501Z layer_outputs = layer_module( 2025-11-03T16:35:43.2172715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2172799Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2173030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2173094Z return func(*args, **kwargs) 2025-11-03T16:35:43.2173330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2173418Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2173654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2173769Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2174002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2174078Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2174082Z 2025-11-03T16:35:43.2174197Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2174397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2174458Z res = mod(**inputs) 2025-11-03T16:35:43.2174697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2174781Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2175011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2175088Z layer_outputs = layer_module( 2025-11-03T16:35:43.2175298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2175381Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2175615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2175687Z return func(*args, **kwargs) 2025-11-03T16:35:43.2175916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2176002Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2176237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2176347Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2176582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2176667Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2176672Z 2025-11-03T16:35:43.2176769Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2176985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2177046Z res = mod(**inputs) 2025-11-03T16:35:43.2177287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2177355Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2177584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2177660Z layer_outputs = layer_module( 2025-11-03T16:35:43.2177874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2177973Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2178202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2178277Z return func(*args, **kwargs) 2025-11-03T16:35:43.2178505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2178592Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2178827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2178936Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2179170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2179246Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2179251Z 2025-11-03T16:35:43.2179350Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2179545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2179607Z res = mod(**inputs) 2025-11-03T16:35:43.2179861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2179930Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2180165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2180232Z layer_outputs = layer_module( 2025-11-03T16:35:43.2180460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2180544Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2180772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2180847Z return func(*args, **kwargs) 2025-11-03T16:35:43.2181074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2181153Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2181389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2181454Z return func(*args, **kwargs) 2025-11-03T16:35:43.2181686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2181768Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2181998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2182069Z return func(*args, **kwargs) 2025-11-03T16:35:43.2182301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2182381Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2182385Z 2025-11-03T16:35:43.2182500Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2182698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2182761Z res = mod(**inputs) 2025-11-03T16:35:43.2182991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2183067Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2183298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2183372Z layer_outputs = layer_module( 2025-11-03T16:35:43.2183585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2183685Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2183922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2183987Z return func(*args, **kwargs) 2025-11-03T16:35:43.2184224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2184302Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2184532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2184605Z return func(*args, **kwargs) 2025-11-03T16:35:43.2184835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2184921Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2185150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2185222Z return func(*args, **kwargs) 2025-11-03T16:35:43.2185451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2185542Z key_states = self.k(current_states) 2025-11-03T16:35:43.2185546Z 2025-11-03T16:35:43.2185652Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2185839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2185909Z res = mod(**inputs) 2025-11-03T16:35:43.2186167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2186238Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2186479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2186548Z layer_outputs = layer_module( 2025-11-03T16:35:43.2186767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2186844Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2187080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2187153Z return func(*args, **kwargs) 2025-11-03T16:35:43.2187380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2187464Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2187694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2187766Z return func(*args, **kwargs) 2025-11-03T16:35:43.2187993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2188082Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2188314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2188395Z return func(*args, **kwargs) 2025-11-03T16:35:43.2188625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2188696Z value_states = self.v(current_states) 2025-11-03T16:35:43.2188699Z 2025-11-03T16:35:43.2188774Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2188857Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2188954Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2189142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2189219Z res = mod(**inputs) 2025-11-03T16:35:43.2189445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2189518Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2189746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2189820Z layer_outputs = layer_module( 2025-11-03T16:35:43.2190034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2190115Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2190348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2190412Z return func(*args, **kwargs) 2025-11-03T16:35:43.2190648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2190726Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2190965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2191031Z return func(*args, **kwargs) 2025-11-03T16:35:43.2191277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2191363Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2191591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2191664Z return func(*args, **kwargs) 2025-11-03T16:35:43.2191904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2191979Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2191991Z 2025-11-03T16:35:43.2192092Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2192282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2192350Z res = mod(**inputs) 2025-11-03T16:35:43.2192586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2192663Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2192896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2192963Z layer_outputs = layer_module( 2025-11-03T16:35:43.2193182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2193257Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2193493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2193560Z return func(*args, **kwargs) 2025-11-03T16:35:43.2193790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2193892Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2194187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2194266Z return func(*args, **kwargs) 2025-11-03T16:35:43.2194493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 488, in forward 2025-11-03T16:35:43.2194620Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.2194633Z 2025-11-03T16:35:43.2194733Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2194932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2195022Z res = mod(**inputs) 2025-11-03T16:35:43.2195362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2195437Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2195665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2195732Z layer_outputs = layer_module( 2025-11-03T16:35:43.2195946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2196018Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2196250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2196313Z return func(*args, **kwargs) 2025-11-03T16:35:43.2196534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2196618Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2196839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2196913Z return func(*args, **kwargs) 2025-11-03T16:35:43.2197158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2197249Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2197475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2197538Z return func(*args, **kwargs) 2025-11-03T16:35:43.2197785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2197858Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2197863Z 2025-11-03T16:35:43.2197970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2198158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2198217Z res = mod(**inputs) 2025-11-03T16:35:43.2198453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2198520Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2198751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2198816Z layer_outputs = layer_module( 2025-11-03T16:35:43.2199025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2199108Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2199328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2199401Z return func(*args, **kwargs) 2025-11-03T16:35:43.2199622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2199728Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2199951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2200014Z return func(*args, **kwargs) 2025-11-03T16:35:43.2200245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2200324Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2200553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2200616Z return func(*args, **kwargs) 2025-11-03T16:35:43.2200854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2200933Z key_states = self.k(current_states) 2025-11-03T16:35:43.2200937Z 2025-11-03T16:35:43.2201035Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2201225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2201285Z res = mod(**inputs) 2025-11-03T16:35:43.2201516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2201583Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2201804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2201877Z layer_outputs = layer_module( 2025-11-03T16:35:43.2202081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2202162Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2202383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2202448Z return func(*args, **kwargs) 2025-11-03T16:35:43.2202690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2202765Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2202997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2203059Z return func(*args, **kwargs) 2025-11-03T16:35:43.2203296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2203382Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2203606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2203674Z return func(*args, **kwargs) 2025-11-03T16:35:43.2203895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2203976Z value_states = self.v(current_states) 2025-11-03T16:35:43.2203979Z 2025-11-03T16:35:43.2204054Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2204129Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2204232Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2204417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2204485Z res = mod(**inputs) 2025-11-03T16:35:43.2204709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2204777Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2205007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2205073Z layer_outputs = layer_module( 2025-11-03T16:35:43.2205316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2205387Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2205611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2205682Z return func(*args, **kwargs) 2025-11-03T16:35:43.2205904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2205986Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2206207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2206286Z return func(*args, **kwargs) 2025-11-03T16:35:43.2206519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2206599Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2206834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2206897Z return func(*args, **kwargs) 2025-11-03T16:35:43.2207131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2207202Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2207206Z 2025-11-03T16:35:43.2207305Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2207500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2207561Z res = mod(**inputs) 2025-11-03T16:35:43.2207798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2207864Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2208136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2208211Z layer_outputs = layer_module( 2025-11-03T16:35:43.2208415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2208494Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2208715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2208792Z return func(*args, **kwargs) 2025-11-03T16:35:43.2209024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2209111Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2209341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2209452Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2209684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2209777Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2209780Z 2025-11-03T16:35:43.2209874Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2210063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2210122Z res = mod(**inputs) 2025-11-03T16:35:43.2210354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2210423Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2210645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2210737Z layer_outputs = layer_module( 2025-11-03T16:35:43.2210945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2211025Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2211251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2211322Z return func(*args, **kwargs) 2025-11-03T16:35:43.2211545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2211628Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2211861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2211987Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2212217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2212294Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2212297Z 2025-11-03T16:35:43.2212390Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2212581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2212641Z res = mod(**inputs) 2025-11-03T16:35:43.2212875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2212942Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2213175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2213367Z layer_outputs = layer_module( 2025-11-03T16:35:43.2213585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2213672Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2213941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2214017Z return func(*args, **kwargs) 2025-11-03T16:35:43.2214243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2214327Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2214588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2214696Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2214935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2215019Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2215022Z 2025-11-03T16:35:43.2215123Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2215316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2215378Z res = mod(**inputs) 2025-11-03T16:35:43.2215614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2215684Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2215918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2215984Z layer_outputs = layer_module( 2025-11-03T16:35:43.2216189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2216272Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2216496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2216593Z return func(*args, **kwargs) 2025-11-03T16:35:43.2216820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2216903Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2217142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2217252Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2217493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2217591Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2217595Z 2025-11-03T16:35:43.2217698Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2217893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2217955Z res = mod(**inputs) 2025-11-03T16:35:43.2218187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2218253Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2218489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2218558Z layer_outputs = layer_module( 2025-11-03T16:35:43.2218773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2218856Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2219091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2219162Z return func(*args, **kwargs) 2025-11-03T16:35:43.2219386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2219479Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2219708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2219775Z return func(*args, **kwargs) 2025-11-03T16:35:43.2220003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2220081Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2220323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2220388Z return func(*args, **kwargs) 2025-11-03T16:35:43.2220608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2220687Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2220692Z 2025-11-03T16:35:43.2220786Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2220977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2221035Z res = mod(**inputs) 2025-11-03T16:35:43.2221260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2221334Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2221558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2221632Z layer_outputs = layer_module( 2025-11-03T16:35:43.2221839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2221912Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2222140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2222220Z return func(*args, **kwargs) 2025-11-03T16:35:43.2222451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2222526Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2222759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2222821Z return func(*args, **kwargs) 2025-11-03T16:35:43.2223045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2223144Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2223371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2223440Z return func(*args, **kwargs) 2025-11-03T16:35:43.2223665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2223735Z key_states = self.k(current_states) 2025-11-03T16:35:43.2223738Z 2025-11-03T16:35:43.2223842Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2224024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2224090Z res = mod(**inputs) 2025-11-03T16:35:43.2224316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2224384Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2224616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2224681Z layer_outputs = layer_module( 2025-11-03T16:35:43.2224896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2224969Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2225217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2225280Z return func(*args, **kwargs) 2025-11-03T16:35:43.2225508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2225592Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2225849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2225921Z return func(*args, **kwargs) 2025-11-03T16:35:43.2226142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2226218Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2226452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2226515Z return func(*args, **kwargs) 2025-11-03T16:35:43.2226747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2226819Z value_states = self.v(current_states) 2025-11-03T16:35:43.2226822Z 2025-11-03T16:35:43.2226903Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2226978Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2227075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2227266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2227328Z res = mod(**inputs) 2025-11-03T16:35:43.2227564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2227648Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2227881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2227955Z layer_outputs = layer_module( 2025-11-03T16:35:43.2228169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2228247Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2228470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2228535Z return func(*args, **kwargs) 2025-11-03T16:35:43.2228763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 564, in forward 2025-11-03T16:35:43.2228854Z self_attention_outputs = self.layer[0]( 2025-11-03T16:35:43.2229083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2229146Z return func(*args, **kwargs) 2025-11-03T16:35:43.2229368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 478, in forward 2025-11-03T16:35:43.2229450Z attention_output = self.SelfAttention( 2025-11-03T16:35:43.2229669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2229738Z return func(*args, **kwargs) 2025-11-03T16:35:43.2229960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2230037Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2230042Z 2025-11-03T16:35:43.2230135Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2230320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2230388Z res = mod(**inputs) 2025-11-03T16:35:43.2230629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2230704Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2230934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2231002Z layer_outputs = layer_module( 2025-11-03T16:35:43.2231241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2231317Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2231554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2231619Z return func(*args, **kwargs) 2025-11-03T16:35:43.2231850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2231936Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2232171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2232242Z return func(*args, **kwargs) 2025-11-03T16:35:43.2232472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2232559Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2232788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2232852Z return func(*args, **kwargs) 2025-11-03T16:35:43.2233088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-11-03T16:35:43.2233162Z query_states = self.q(hidden_states) 2025-11-03T16:35:43.2233166Z 2025-11-03T16:35:43.2233286Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2233482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2233542Z res = mod(**inputs) 2025-11-03T16:35:43.2233787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2233855Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2234197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2234271Z layer_outputs = layer_module( 2025-11-03T16:35:43.2234488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2234591Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2234834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2234919Z return func(*args, **kwargs) 2025-11-03T16:35:43.2235151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2235239Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2235474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2235538Z return func(*args, **kwargs) 2025-11-03T16:35:43.2235781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2235861Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2236102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2236169Z return func(*args, **kwargs) 2025-11-03T16:35:43.2236399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 387, in forward 2025-11-03T16:35:43.2236499Z key_states = self.k(current_states) 2025-11-03T16:35:43.2236503Z 2025-11-03T16:35:43.2236603Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2236801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2236865Z res = mod(**inputs) 2025-11-03T16:35:43.2237114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2237191Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2237424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2237500Z layer_outputs = layer_module( 2025-11-03T16:35:43.2237714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2237797Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2238031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2238096Z return func(*args, **kwargs) 2025-11-03T16:35:43.2238338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2238414Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2238654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2238719Z return func(*args, **kwargs) 2025-11-03T16:35:43.2238950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2239041Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2239271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2239361Z return func(*args, **kwargs) 2025-11-03T16:35:43.2239593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-11-03T16:35:43.2239674Z value_states = self.v(current_states) 2025-11-03T16:35:43.2239677Z 2025-11-03T16:35:43.2239755Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2239832Z cudagraph partition due to non gpu ops 2025-11-03T16:35:43.2239941Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2240133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2240221Z res = mod(**inputs) 2025-11-03T16:35:43.2240457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2240525Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2240771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2240845Z layer_outputs = layer_module( 2025-11-03T16:35:43.2241069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2241144Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2241380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2241455Z return func(*args, **kwargs) 2025-11-03T16:35:43.2241687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2241775Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2242008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2242076Z return func(*args, **kwargs) 2025-11-03T16:35:43.2242325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 516, in forward 2025-11-03T16:35:43.2242405Z attention_output = self.EncDecAttention( 2025-11-03T16:35:43.2242640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2242704Z return func(*args, **kwargs) 2025-11-03T16:35:43.2242955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 446, in forward 2025-11-03T16:35:43.2243029Z attn_output = self.o(attn_output) 2025-11-03T16:35:43.2243034Z 2025-11-03T16:35:43.2243133Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2243329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2243391Z res = mod(**inputs) 2025-11-03T16:35:43.2243639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2243708Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2243941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2244018Z layer_outputs = layer_module( 2025-11-03T16:35:43.2244232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2244315Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2244548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2244615Z return func(*args, **kwargs) 2025-11-03T16:35:43.2244856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 588, in forward 2025-11-03T16:35:43.2244952Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:35:43.2245190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2245255Z return func(*args, **kwargs) 2025-11-03T16:35:43.2245490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 528, in forward 2025-11-03T16:35:43.2245615Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:35:43.2245620Z 2025-11-03T16:35:43.2245718Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2245914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2245992Z res = mod(**inputs) 2025-11-03T16:35:43.2246233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2246303Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2246539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2246615Z layer_outputs = layer_module( 2025-11-03T16:35:43.2246827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2246908Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2247142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2247214Z return func(*args, **kwargs) 2025-11-03T16:35:43.2247444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2247532Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2247764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2247902Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2248137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-11-03T16:35:43.2248232Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-11-03T16:35:43.2248235Z 2025-11-03T16:35:43.2248331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2248538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2248601Z res = mod(**inputs) 2025-11-03T16:35:43.2248837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2248906Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2249131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2249209Z layer_outputs = layer_module( 2025-11-03T16:35:43.2249421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2249503Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2249732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2249807Z return func(*args, **kwargs) 2025-11-03T16:35:43.2250036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2250123Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2250361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2250467Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2250719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-11-03T16:35:43.2250795Z hidden_linear = self.wi_1(hidden_states) 2025-11-03T16:35:43.2250799Z 2025-11-03T16:35:43.2250894Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2251088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2251149Z res = mod(**inputs) 2025-11-03T16:35:43.2251385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2251452Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2251691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2251776Z layer_outputs = layer_module( 2025-11-03T16:35:43.2251991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2252074Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2252308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2252382Z return func(*args, **kwargs) 2025-11-03T16:35:43.2252612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2252697Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2252937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2253047Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2253284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-11-03T16:35:43.2253367Z hidden_states = hidden_gelu * hidden_linear 2025-11-03T16:35:43.2253372Z 2025-11-03T16:35:43.2253476Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2253682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2253745Z res = mod(**inputs) 2025-11-03T16:35:43.2253984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1786, in forward 2025-11-03T16:35:43.2254051Z decoder_outputs = self.decoder( 2025-11-03T16:35:43.2254303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1086, in forward 2025-11-03T16:35:43.2254372Z layer_outputs = layer_module( 2025-11-03T16:35:43.2254585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:35:43.2254672Z return super().__call__(*args, **kwargs) 2025-11-03T16:35:43.2254906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:35:43.2254982Z return func(*args, **kwargs) 2025-11-03T16:35:43.2255211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 614, in forward 2025-11-03T16:35:43.2255298Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:35:43.2255535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-11-03T16:35:43.2255646Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:35:43.2255881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-11-03T16:35:43.2255958Z hidden_states = self.wo(hidden_states) 2025-11-03T16:35:43.2255961Z 2025-11-03T16:35:43.2256066Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2256253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2256330Z res = mod(**inputs) 2025-11-03T16:35:43.2256575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1815, in forward 2025-11-03T16:35:43.2256657Z lm_logits = self.lm_head(sequence_output) 2025-11-03T16:35:43.2256661Z 2025-11-03T16:35:43.2256767Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:35:43.2256958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:35:43.2257019Z res = mod(**inputs) 2025-11-03T16:35:43.2257263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1822, in forward 2025-11-03T16:35:43.2257415Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-11-03T16:35:43.2257419Z 2025-11-03T16:35:52.9891548Z Compilation time (from dynamo_timed): 20.32003645 2025-11-03T16:35:53.0060679Z pass 2025-11-03T16:35:53.0065096Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:35:53.0069043Z TIMING: _recursive_pre_grad_passes:0.01378 _recursive_joint_graph_passes:0.76389 _recursive_post_grad_passes:0.08433 async_compile.wait:0.64542 code_gen:9.10978 inductor_compile:11.28286 backend_compile:16.2793 gc:0.00124 entire_frame_compile:20.32004 total_wall_time:20.32004 2025-11-03T16:35:53.0070557Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:16283 | FakeTensor.__torch_dispatch__:6387 | ProxyTorchDispatchMode.__torch_dispatch__:4865 2025-11-03T16:35:53.0071105Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-11-03T16:35:55.3822191Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:35:55.3823106Z import pynvml # type: ignore[import] 2025-11-03T16:35:58.4565224Z 2025-11-03T16:35:58.4684085Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-11-03T16:35:58.4689234Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-11-03T16:36:01.0474466Z 2025-11-03T16:36:01.0479354Z loading model: 0it [00:02, ?it/s] 2025-11-03T16:36:01.0500252Z cpu eval MegatronBertForCausalLM 2025-11-03T16:36:02.6032718Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:36:03.1592224Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:36:03.7081655Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:36:17.4319327Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4320037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4321012Z res = mod(**inputs) 2025-11-03T16:36:17.4321714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4322305Z outputs = self.bert( 2025-11-03T16:36:17.4323291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4323924Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4324412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4324851Z layer_outputs = layer_module( 2025-11-03T16:36:17.4325218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4325990Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4326426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4326855Z return func(*args, **kwargs) 2025-11-03T16:36:17.4327319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4327807Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4328257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4328701Z return func(*args, **kwargs) 2025-11-03T16:36:17.4329123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4329580Z self_outputs = self.self( 2025-11-03T16:36:17.4329978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4330381Z return func(*args, **kwargs) 2025-11-03T16:36:17.4330842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4331325Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4331474Z 2025-11-03T16:36:17.4331599Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4331982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4332317Z res = mod(**inputs) 2025-11-03T16:36:17.4332742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4333160Z outputs = self.bert( 2025-11-03T16:36:17.4333612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4334040Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4334443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4334857Z layer_outputs = layer_module( 2025-11-03T16:36:17.4335263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4335629Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4335996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4336376Z return func(*args, **kwargs) 2025-11-03T16:36:17.4336848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4337288Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4337676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4338045Z return func(*args, **kwargs) 2025-11-03T16:36:17.4338460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4338892Z self_outputs = self.self( 2025-11-03T16:36:17.4339292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4339662Z return func(*args, **kwargs) 2025-11-03T16:36:17.4340061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4340481Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4340650Z 2025-11-03T16:36:17.4340756Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4341124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4341437Z res = mod(**inputs) 2025-11-03T16:36:17.4341843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4342264Z outputs = self.bert( 2025-11-03T16:36:17.4342672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4343111Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4343548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4343973Z layer_outputs = layer_module( 2025-11-03T16:36:17.4344329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4344699Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4345082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4345451Z return func(*args, **kwargs) 2025-11-03T16:36:17.4345861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4346297Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4346682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4347056Z return func(*args, **kwargs) 2025-11-03T16:36:17.4347458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4347882Z self_outputs = self.self( 2025-11-03T16:36:17.4348270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4348643Z return func(*args, **kwargs) 2025-11-03T16:36:17.4349046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4349478Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4349621Z 2025-11-03T16:36:17.4349730Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4349954Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4350199Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4350555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4350888Z res = mod(**inputs) 2025-11-03T16:36:17.4351293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4351716Z outputs = self.bert( 2025-11-03T16:36:17.4352106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4352531Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4352950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4353375Z layer_outputs = layer_module( 2025-11-03T16:36:17.4353727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4354217Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4354634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4355072Z return func(*args, **kwargs) 2025-11-03T16:36:17.4355485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4355920Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4356299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4356673Z return func(*args, **kwargs) 2025-11-03T16:36:17.4357089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4357571Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4358068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4358507Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4358661Z 2025-11-03T16:36:17.4358767Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4359133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4359467Z res = mod(**inputs) 2025-11-03T16:36:17.4359864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4360289Z outputs = self.bert( 2025-11-03T16:36:17.4360696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4361117Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4361537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4361957Z layer_outputs = layer_module( 2025-11-03T16:36:17.4362312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4362722Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4363107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4363487Z return func(*args, **kwargs) 2025-11-03T16:36:17.4363896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4364365Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4364779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4365190Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4365641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4366134Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4366611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4367048Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4367195Z 2025-11-03T16:36:17.4367305Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4367655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4367973Z res = mod(**inputs) 2025-11-03T16:36:17.4368368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4368791Z outputs = self.bert( 2025-11-03T16:36:17.4369192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4369635Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4370063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4370487Z layer_outputs = layer_module( 2025-11-03T16:36:17.4370842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4371210Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4371591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4371988Z return func(*args, **kwargs) 2025-11-03T16:36:17.4372404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4372848Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4373265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4373700Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4374189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4374714Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4375205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4375669Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4376073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4376426Z return self.act(input) 2025-11-03T16:36:17.4376537Z 2025-11-03T16:36:17.4376653Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4377071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4377396Z res = mod(**inputs) 2025-11-03T16:36:17.4377807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4378221Z outputs = self.bert( 2025-11-03T16:36:17.4378624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4379072Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4379518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4379946Z layer_outputs = layer_module( 2025-11-03T16:36:17.4380302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4380672Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4381050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4381429Z return func(*args, **kwargs) 2025-11-03T16:36:17.4381842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4382292Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4382714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4383163Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4383703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4384239Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4384735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4385176Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4385323Z 2025-11-03T16:36:17.4385429Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4385799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4386133Z res = mod(**inputs) 2025-11-03T16:36:17.4386546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4386999Z outputs = self.bert( 2025-11-03T16:36:17.4387395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4387819Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4388248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4388665Z layer_outputs = layer_module( 2025-11-03T16:36:17.4389013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4389380Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4389767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4390178Z return func(*args, **kwargs) 2025-11-03T16:36:17.4390635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4391141Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4391537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4391934Z return func(*args, **kwargs) 2025-11-03T16:36:17.4392397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4392855Z self_outputs = self.self( 2025-11-03T16:36:17.4393246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4393668Z return func(*args, **kwargs) 2025-11-03T16:36:17.4394235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4394754Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4394909Z 2025-11-03T16:36:17.4395031Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4395418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4395767Z res = mod(**inputs) 2025-11-03T16:36:17.4396219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4396656Z outputs = self.bert( 2025-11-03T16:36:17.4397054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4397484Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4397914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4398342Z layer_outputs = layer_module( 2025-11-03T16:36:17.4398697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4399065Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4399451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4399807Z return func(*args, **kwargs) 2025-11-03T16:36:17.4400201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4400616Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4400978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4401335Z return func(*args, **kwargs) 2025-11-03T16:36:17.4416853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4417567Z self_outputs = self.self( 2025-11-03T16:36:17.4417955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4418326Z return func(*args, **kwargs) 2025-11-03T16:36:17.4418740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4419179Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4419315Z 2025-11-03T16:36:17.4419429Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4419800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4420129Z res = mod(**inputs) 2025-11-03T16:36:17.4420536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4420958Z outputs = self.bert( 2025-11-03T16:36:17.4421355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4421780Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4422231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4422637Z layer_outputs = layer_module( 2025-11-03T16:36:17.4422984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4423345Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4423751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4424127Z return func(*args, **kwargs) 2025-11-03T16:36:17.4424523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4424937Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4425321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4425685Z return func(*args, **kwargs) 2025-11-03T16:36:17.4426081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4426482Z self_outputs = self.self( 2025-11-03T16:36:17.4426826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4427182Z return func(*args, **kwargs) 2025-11-03T16:36:17.4427573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4427994Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4428132Z 2025-11-03T16:36:17.4428224Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4428437Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4428727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4429119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4429445Z res = mod(**inputs) 2025-11-03T16:36:17.4429832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4430245Z outputs = self.bert( 2025-11-03T16:36:17.4430644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4431067Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4431537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4431958Z layer_outputs = layer_module( 2025-11-03T16:36:17.4432323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4432706Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4433095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4433470Z return func(*args, **kwargs) 2025-11-03T16:36:17.4433878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4434422Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4434835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4435299Z return func(*args, **kwargs) 2025-11-03T16:36:17.4435743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4436244Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4436756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4437184Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4437324Z 2025-11-03T16:36:17.4437438Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4437809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4438132Z res = mod(**inputs) 2025-11-03T16:36:17.4438532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4438945Z outputs = self.bert( 2025-11-03T16:36:17.4439346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4439759Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4440182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4440618Z layer_outputs = layer_module( 2025-11-03T16:36:17.4440974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4441340Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4441713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4442078Z return func(*args, **kwargs) 2025-11-03T16:36:17.4442485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4442915Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4443334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4443726Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4444172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4444654Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4445110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4445506Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4445674Z 2025-11-03T16:36:17.4445778Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4446128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4446453Z res = mod(**inputs) 2025-11-03T16:36:17.4446836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4447237Z outputs = self.bert( 2025-11-03T16:36:17.4447625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4448034Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4448484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4448892Z layer_outputs = layer_module( 2025-11-03T16:36:17.4449233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4449593Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4449964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4450330Z return func(*args, **kwargs) 2025-11-03T16:36:17.4450786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4451195Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4451588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4451977Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4452427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4452885Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4453332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4453790Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4454171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4454502Z return self.act(input) 2025-11-03T16:36:17.4454610Z 2025-11-03T16:36:17.4454710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4455060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4455373Z res = mod(**inputs) 2025-11-03T16:36:17.4455758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4456162Z outputs = self.bert( 2025-11-03T16:36:17.4456551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4456954Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4457376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4457781Z layer_outputs = layer_module( 2025-11-03T16:36:17.4458114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4458471Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4458844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4459212Z return func(*args, **kwargs) 2025-11-03T16:36:17.4459623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4460054Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4460474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4460870Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4461314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4461828Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4462302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4462722Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4462867Z 2025-11-03T16:36:17.4462973Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4463332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4463640Z res = mod(**inputs) 2025-11-03T16:36:17.4464033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4464462Z outputs = self.bert( 2025-11-03T16:36:17.4464859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4465293Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4465715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4466161Z layer_outputs = layer_module( 2025-11-03T16:36:17.4466525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4466888Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4467272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4467641Z return func(*args, **kwargs) 2025-11-03T16:36:17.4468050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4468517Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4468953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4469371Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4469839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4470364Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4470853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4471288Z return input_tensor + hidden_states 2025-11-03T16:36:17.4471442Z 2025-11-03T16:36:17.4471551Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4471922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4472248Z res = mod(**inputs) 2025-11-03T16:36:17.4472649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4473056Z outputs = self.bert( 2025-11-03T16:36:17.4473459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4473906Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4474448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4474893Z layer_outputs = layer_module( 2025-11-03T16:36:17.4475269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4475664Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4476088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4476461Z return func(*args, **kwargs) 2025-11-03T16:36:17.4476870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4477292Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4477674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4478050Z return func(*args, **kwargs) 2025-11-03T16:36:17.4478456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4478866Z self_outputs = self.self( 2025-11-03T16:36:17.4479250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4479614Z return func(*args, **kwargs) 2025-11-03T16:36:17.4480015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4480438Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4480595Z 2025-11-03T16:36:17.4480699Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4481055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4481372Z res = mod(**inputs) 2025-11-03T16:36:17.4481770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4482168Z outputs = self.bert( 2025-11-03T16:36:17.4482546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4482953Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4483358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4483760Z layer_outputs = layer_module( 2025-11-03T16:36:17.4484094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4484439Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4484804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4485162Z return func(*args, **kwargs) 2025-11-03T16:36:17.4485553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4485982Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4486349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4486704Z return func(*args, **kwargs) 2025-11-03T16:36:17.4487107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4487511Z self_outputs = self.self( 2025-11-03T16:36:17.4487850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4488222Z return func(*args, **kwargs) 2025-11-03T16:36:17.4488620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4489030Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4489157Z 2025-11-03T16:36:17.4489257Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4489603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4489911Z res = mod(**inputs) 2025-11-03T16:36:17.4490291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4490687Z outputs = self.bert( 2025-11-03T16:36:17.4491063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4491464Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4491865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4492265Z layer_outputs = layer_module( 2025-11-03T16:36:17.4492616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4492960Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4493322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4493675Z return func(*args, **kwargs) 2025-11-03T16:36:17.4494078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4494475Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4494843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4495187Z return func(*args, **kwargs) 2025-11-03T16:36:17.4495568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4495958Z self_outputs = self.self( 2025-11-03T16:36:17.4496287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4496627Z return func(*args, **kwargs) 2025-11-03T16:36:17.4497003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4497409Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4497537Z 2025-11-03T16:36:17.4497623Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4497823Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4498051Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4498398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4498701Z res = mod(**inputs) 2025-11-03T16:36:17.4499095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4499488Z outputs = self.bert( 2025-11-03T16:36:17.4499877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4500282Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4500682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4501070Z layer_outputs = layer_module( 2025-11-03T16:36:17.4501403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4501762Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4502128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4502486Z return func(*args, **kwargs) 2025-11-03T16:36:17.4502873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4503288Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4503660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4504017Z return func(*args, **kwargs) 2025-11-03T16:36:17.4504406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4504864Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4505320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4505737Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4505869Z 2025-11-03T16:36:17.4505992Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4506329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4506634Z res = mod(**inputs) 2025-11-03T16:36:17.4507016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4507418Z outputs = self.bert( 2025-11-03T16:36:17.4507818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4508209Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4508605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4509006Z layer_outputs = layer_module( 2025-11-03T16:36:17.4509340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4509680Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4510036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4510398Z return func(*args, **kwargs) 2025-11-03T16:36:17.4510788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4511197Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4511577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4511954Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4512379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4512856Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4513453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4513879Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4514068Z 2025-11-03T16:36:17.4514180Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4514546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4514867Z res = mod(**inputs) 2025-11-03T16:36:17.4515317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4515727Z outputs = self.bert( 2025-11-03T16:36:17.4516186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4516594Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4516999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4517395Z layer_outputs = layer_module( 2025-11-03T16:36:17.4517737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4518094Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4518463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4518827Z return func(*args, **kwargs) 2025-11-03T16:36:17.4519222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4519643Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4520059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4520442Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4520873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4521324Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4521784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4522230Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4522602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4522933Z return self.act(input) 2025-11-03T16:36:17.4523042Z 2025-11-03T16:36:17.4523140Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4523486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4523797Z res = mod(**inputs) 2025-11-03T16:36:17.4524180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4524571Z outputs = self.bert( 2025-11-03T16:36:17.4524961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4525360Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4525763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4526165Z layer_outputs = layer_module( 2025-11-03T16:36:17.4526490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4526869Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4527236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4527601Z return func(*args, **kwargs) 2025-11-03T16:36:17.4527993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4528400Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4528788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4529184Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4530178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4530668Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4531123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4531540Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4531679Z 2025-11-03T16:36:17.4531776Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4532123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4532431Z res = mod(**inputs) 2025-11-03T16:36:17.4532808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4533203Z outputs = self.bert( 2025-11-03T16:36:17.4533586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4533986Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4534394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4534790Z layer_outputs = layer_module( 2025-11-03T16:36:17.4535115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4535459Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4535863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4536225Z return func(*args, **kwargs) 2025-11-03T16:36:17.4536609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4537016Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4537383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4537732Z return func(*args, **kwargs) 2025-11-03T16:36:17.4538117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4538505Z self_outputs = self.self( 2025-11-03T16:36:17.4538855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4539205Z return func(*args, **kwargs) 2025-11-03T16:36:17.4539587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4539996Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4540127Z 2025-11-03T16:36:17.4540256Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4540663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4540975Z res = mod(**inputs) 2025-11-03T16:36:17.4541358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4541759Z outputs = self.bert( 2025-11-03T16:36:17.4542139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4542543Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4542952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4543360Z layer_outputs = layer_module( 2025-11-03T16:36:17.4543687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4544040Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4544402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4544756Z return func(*args, **kwargs) 2025-11-03T16:36:17.4545147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4545550Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4545907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4546248Z return func(*args, **kwargs) 2025-11-03T16:36:17.4546633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4547023Z self_outputs = self.self( 2025-11-03T16:36:17.4547358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4547716Z return func(*args, **kwargs) 2025-11-03T16:36:17.4548102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4548510Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4548636Z 2025-11-03T16:36:17.4548736Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4549134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4549445Z res = mod(**inputs) 2025-11-03T16:36:17.4549834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4550239Z outputs = self.bert( 2025-11-03T16:36:17.4550615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4551024Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4551429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4551833Z layer_outputs = layer_module( 2025-11-03T16:36:17.4552166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4552512Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4552894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4553257Z return func(*args, **kwargs) 2025-11-03T16:36:17.4553647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4554137Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4554532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4554902Z return func(*args, **kwargs) 2025-11-03T16:36:17.4555314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4555717Z self_outputs = self.self( 2025-11-03T16:36:17.4556061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4556418Z return func(*args, **kwargs) 2025-11-03T16:36:17.4556809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4557238Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4557367Z 2025-11-03T16:36:17.4557455Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4557657Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4557880Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4558221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4558525Z res = mod(**inputs) 2025-11-03T16:36:17.4558894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4559281Z outputs = self.bert( 2025-11-03T16:36:17.4559659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4560057Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4560452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4560841Z layer_outputs = layer_module( 2025-11-03T16:36:17.4561188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4561535Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4561894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4562241Z return func(*args, **kwargs) 2025-11-03T16:36:17.4562653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4563070Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4563441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4563808Z return func(*args, **kwargs) 2025-11-03T16:36:17.4564202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4564660Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4565112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4565532Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4565664Z 2025-11-03T16:36:17.4565773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4566111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4566423Z res = mod(**inputs) 2025-11-03T16:36:17.4566806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4567213Z outputs = self.bert( 2025-11-03T16:36:17.4567600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4568010Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4568409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4568810Z layer_outputs = layer_module( 2025-11-03T16:36:17.4569143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4569482Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4569836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4570209Z return func(*args, **kwargs) 2025-11-03T16:36:17.4570616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4571033Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4571416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4571793Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4572221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4572681Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4573113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4573519Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4573658Z 2025-11-03T16:36:17.4573759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4574108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4574422Z res = mod(**inputs) 2025-11-03T16:36:17.4574817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4575210Z outputs = self.bert( 2025-11-03T16:36:17.4575581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4575976Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4576387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4576789Z layer_outputs = layer_module( 2025-11-03T16:36:17.4577122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4577479Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4577850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4578220Z return func(*args, **kwargs) 2025-11-03T16:36:17.4578613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4579044Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4579428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4579807Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4580234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4580695Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4581126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4581579Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4581943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4582271Z return self.act(input) 2025-11-03T16:36:17.4582377Z 2025-11-03T16:36:17.4582476Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4582823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4583145Z res = mod(**inputs) 2025-11-03T16:36:17.4583531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4583944Z outputs = self.bert( 2025-11-03T16:36:17.4584321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4584728Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4585129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4585529Z layer_outputs = layer_module( 2025-11-03T16:36:17.4585857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4586207Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4586572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4586930Z return func(*args, **kwargs) 2025-11-03T16:36:17.4587316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4587734Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4588124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4588529Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4588959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4589441Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4589901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4590316Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4590452Z 2025-11-03T16:36:17.4590560Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4590904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4591205Z res = mod(**inputs) 2025-11-03T16:36:17.4591586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4591982Z outputs = self.bert( 2025-11-03T16:36:17.4592364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4592769Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4593163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4593565Z layer_outputs = layer_module( 2025-11-03T16:36:17.4593905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4594339Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4594768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4595169Z return func(*args, **kwargs) 2025-11-03T16:36:17.4595577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4595983Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4596359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4596721Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4597145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4597639Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4598082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4598479Z return input_tensor + hidden_states 2025-11-03T16:36:17.4598602Z 2025-11-03T16:36:17.4598702Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4599037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4599337Z res = mod(**inputs) 2025-11-03T16:36:17.4599713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4600101Z outputs = self.bert( 2025-11-03T16:36:17.4600463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4600855Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4601245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4601641Z layer_outputs = layer_module( 2025-11-03T16:36:17.4601995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4602339Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4602698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4603047Z return func(*args, **kwargs) 2025-11-03T16:36:17.4603446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4603841Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4604206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4604555Z return func(*args, **kwargs) 2025-11-03T16:36:17.4604940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4605336Z self_outputs = self.self( 2025-11-03T16:36:17.4605671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4606018Z return func(*args, **kwargs) 2025-11-03T16:36:17.4606397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4606800Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4606932Z 2025-11-03T16:36:17.4607035Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4607367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4607670Z res = mod(**inputs) 2025-11-03T16:36:17.4608046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4608453Z outputs = self.bert( 2025-11-03T16:36:17.4608816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4609210Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4609607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4610003Z layer_outputs = layer_module( 2025-11-03T16:36:17.4610328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4610680Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4611031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4611376Z return func(*args, **kwargs) 2025-11-03T16:36:17.4611756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4612158Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4612505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4612851Z return func(*args, **kwargs) 2025-11-03T16:36:17.4613397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4613803Z self_outputs = self.self( 2025-11-03T16:36:17.4614136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4614485Z return func(*args, **kwargs) 2025-11-03T16:36:17.4614872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4615281Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4615454Z 2025-11-03T16:36:17.4615563Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4615894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4616201Z res = mod(**inputs) 2025-11-03T16:36:17.4616602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4616989Z outputs = self.bert( 2025-11-03T16:36:17.4617360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4617750Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4618141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4618539Z layer_outputs = layer_module( 2025-11-03T16:36:17.4618875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4619211Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4619568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4619914Z return func(*args, **kwargs) 2025-11-03T16:36:17.4620295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4620694Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4621045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4621392Z return func(*args, **kwargs) 2025-11-03T16:36:17.4621779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4622200Z self_outputs = self.self( 2025-11-03T16:36:17.4622546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4622883Z return func(*args, **kwargs) 2025-11-03T16:36:17.4623261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4623657Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4623782Z 2025-11-03T16:36:17.4623866Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4624088Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4624317Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4624656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4624962Z res = mod(**inputs) 2025-11-03T16:36:17.4625339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4625722Z outputs = self.bert( 2025-11-03T16:36:17.4626095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4626486Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4626879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4627270Z layer_outputs = layer_module( 2025-11-03T16:36:17.4627592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4627930Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4628285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4628652Z return func(*args, **kwargs) 2025-11-03T16:36:17.4629025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4629426Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4629781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4630140Z return func(*args, **kwargs) 2025-11-03T16:36:17.4630523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4630966Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4631404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4631811Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4631942Z 2025-11-03T16:36:17.4632048Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4632384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4632678Z res = mod(**inputs) 2025-11-03T16:36:17.4633057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4633460Z outputs = self.bert( 2025-11-03T16:36:17.4633848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4634317Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4634733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4635174Z layer_outputs = layer_module( 2025-11-03T16:36:17.4635525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4635892Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4636284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4636650Z return func(*args, **kwargs) 2025-11-03T16:36:17.4637048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4637468Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4637894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4638270Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4638716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4639185Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4639621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4640035Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4640166Z 2025-11-03T16:36:17.4640267Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4640615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4640930Z res = mod(**inputs) 2025-11-03T16:36:17.4641315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4641710Z outputs = self.bert( 2025-11-03T16:36:17.4642110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4642515Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4642915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4643316Z layer_outputs = layer_module( 2025-11-03T16:36:17.4643659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4644008Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4644376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4644738Z return func(*args, **kwargs) 2025-11-03T16:36:17.4645127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4645537Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4645925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4646314Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4646737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4647209Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4647623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4648053Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4648413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4648752Z return self.act(input) 2025-11-03T16:36:17.4648856Z 2025-11-03T16:36:17.4648955Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4649295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4649599Z res = mod(**inputs) 2025-11-03T16:36:17.4649971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4650361Z outputs = self.bert( 2025-11-03T16:36:17.4650724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4651135Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4651537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4651940Z layer_outputs = layer_module( 2025-11-03T16:36:17.4652278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4652621Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4652986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4653343Z return func(*args, **kwargs) 2025-11-03T16:36:17.4653736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4654144Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4654534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4654913Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4655342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4655850Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4656287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4656701Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4656839Z 2025-11-03T16:36:17.4656937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4657291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4657600Z res = mod(**inputs) 2025-11-03T16:36:17.4657970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4658362Z outputs = self.bert( 2025-11-03T16:36:17.4658735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4659139Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4659530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4659917Z layer_outputs = layer_module( 2025-11-03T16:36:17.4660250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4660595Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4660951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4661301Z return func(*args, **kwargs) 2025-11-03T16:36:17.4661683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4662095Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4662457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4662827Z return func(*args, **kwargs) 2025-11-03T16:36:17.4663213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4663608Z self_outputs = self.self( 2025-11-03T16:36:17.4663956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4664309Z return func(*args, **kwargs) 2025-11-03T16:36:17.4664729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4665128Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4665264Z 2025-11-03T16:36:17.4665362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4665701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4666006Z res = mod(**inputs) 2025-11-03T16:36:17.4666386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4666784Z outputs = self.bert( 2025-11-03T16:36:17.4667167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4667566Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4667961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4668357Z layer_outputs = layer_module( 2025-11-03T16:36:17.4668691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4669055Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4669425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4669788Z return func(*args, **kwargs) 2025-11-03T16:36:17.4670181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4670608Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4670982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4671339Z return func(*args, **kwargs) 2025-11-03T16:36:17.4671728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4672119Z self_outputs = self.self( 2025-11-03T16:36:17.4672504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4672862Z return func(*args, **kwargs) 2025-11-03T16:36:17.4673255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4673661Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4673796Z 2025-11-03T16:36:17.4673899Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4674311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4674623Z res = mod(**inputs) 2025-11-03T16:36:17.4675013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4675401Z outputs = self.bert( 2025-11-03T16:36:17.4675808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4676212Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4676609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4677021Z layer_outputs = layer_module( 2025-11-03T16:36:17.4677352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4677703Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4678069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4678453Z return func(*args, **kwargs) 2025-11-03T16:36:17.4678840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4679257Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4679631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4679989Z return func(*args, **kwargs) 2025-11-03T16:36:17.4680381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4680777Z self_outputs = self.self( 2025-11-03T16:36:17.4681131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4681488Z return func(*args, **kwargs) 2025-11-03T16:36:17.4681880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4682292Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4682423Z 2025-11-03T16:36:17.4682503Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4682732Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4682965Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4683317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4683618Z res = mod(**inputs) 2025-11-03T16:36:17.4684020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4684425Z outputs = self.bert( 2025-11-03T16:36:17.4684810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4685220Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4685622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4686027Z layer_outputs = layer_module( 2025-11-03T16:36:17.4686377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4686744Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4687108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4687473Z return func(*args, **kwargs) 2025-11-03T16:36:17.4687877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4688291Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4688665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4689022Z return func(*args, **kwargs) 2025-11-03T16:36:17.4689423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4689910Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4690369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4690783Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4690918Z 2025-11-03T16:36:17.4691021Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4691369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4691699Z res = mod(**inputs) 2025-11-03T16:36:17.4692087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4692486Z outputs = self.bert( 2025-11-03T16:36:17.4692870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4693281Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4693686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4694090Z layer_outputs = layer_module( 2025-11-03T16:36:17.4694421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4694769Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4695136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4695501Z return func(*args, **kwargs) 2025-11-03T16:36:17.4695892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4696310Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4696729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4697115Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4697551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4698036Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4698467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4698894Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4699039Z 2025-11-03T16:36:17.4699143Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4699497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4699816Z res = mod(**inputs) 2025-11-03T16:36:17.4700205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4700613Z outputs = self.bert( 2025-11-03T16:36:17.4701006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4701420Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4701822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4702234Z layer_outputs = layer_module( 2025-11-03T16:36:17.4702581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4702939Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4703331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4703685Z return func(*args, **kwargs) 2025-11-03T16:36:17.4704085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4704506Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4704898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4705273Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4705694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4706177Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4706613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4707066Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4707435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4707764Z return self.act(input) 2025-11-03T16:36:17.4707881Z 2025-11-03T16:36:17.4707980Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4708328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4708639Z res = mod(**inputs) 2025-11-03T16:36:17.4709017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4709419Z outputs = self.bert( 2025-11-03T16:36:17.4709801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4710225Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4710627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4711027Z layer_outputs = layer_module( 2025-11-03T16:36:17.4711358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4711722Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4712088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4712446Z return func(*args, **kwargs) 2025-11-03T16:36:17.4712831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4713441Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4713852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4714288Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4714738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4715243Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4715706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4716127Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4716261Z 2025-11-03T16:36:17.4716367Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4716721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4717188Z res = mod(**inputs) 2025-11-03T16:36:17.4717589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4718065Z outputs = self.bert( 2025-11-03T16:36:17.4718459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4718870Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4719285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4719703Z layer_outputs = layer_module( 2025-11-03T16:36:17.4720089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4720449Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4720825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4721213Z return func(*args, **kwargs) 2025-11-03T16:36:17.4721625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4722055Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4722447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4722843Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4723288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4723793Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4724271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4724728Z return input_tensor + hidden_states 2025-11-03T16:36:17.4724868Z 2025-11-03T16:36:17.4724971Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4725324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4725651Z res = mod(**inputs) 2025-11-03T16:36:17.4726054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4726433Z outputs = self.bert( 2025-11-03T16:36:17.4726801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4727194Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4727587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4727981Z layer_outputs = layer_module( 2025-11-03T16:36:17.4728301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4728638Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4728995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4729345Z return func(*args, **kwargs) 2025-11-03T16:36:17.4729721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4730125Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4730480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4730828Z return func(*args, **kwargs) 2025-11-03T16:36:17.4731263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4731646Z self_outputs = self.self( 2025-11-03T16:36:17.4731985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4732058Z return func(*args, **kwargs) 2025-11-03T16:36:17.4732324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4732402Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4732406Z 2025-11-03T16:36:17.4732510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4732714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4732781Z res = mod(**inputs) 2025-11-03T16:36:17.4733050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4733117Z outputs = self.bert( 2025-11-03T16:36:17.4733391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4733461Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4733735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4733803Z layer_outputs = layer_module( 2025-11-03T16:36:17.4734028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4734104Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4734329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4734404Z return func(*args, **kwargs) 2025-11-03T16:36:17.4734683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4734771Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4734995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4735057Z return func(*args, **kwargs) 2025-11-03T16:36:17.4735370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4735439Z self_outputs = self.self( 2025-11-03T16:36:17.4735676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4735741Z return func(*args, **kwargs) 2025-11-03T16:36:17.4736022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4736104Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4736107Z 2025-11-03T16:36:17.4736208Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4736406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4736468Z res = mod(**inputs) 2025-11-03T16:36:17.4736754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4736816Z outputs = self.bert( 2025-11-03T16:36:17.4737089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4737169Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4737439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4737534Z layer_outputs = layer_module( 2025-11-03T16:36:17.4737745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4737828Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4738064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4738133Z return func(*args, **kwargs) 2025-11-03T16:36:17.4738414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4738509Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4738751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4738818Z return func(*args, **kwargs) 2025-11-03T16:36:17.4739096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4739173Z self_outputs = self.self( 2025-11-03T16:36:17.4739404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4739475Z return func(*args, **kwargs) 2025-11-03T16:36:17.4739753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4739831Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4739842Z 2025-11-03T16:36:17.4739920Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4739998Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4740103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4740294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4740366Z res = mod(**inputs) 2025-11-03T16:36:17.4740664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4740730Z outputs = self.bert( 2025-11-03T16:36:17.4741007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4741090Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4741374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4741445Z layer_outputs = layer_module( 2025-11-03T16:36:17.4741656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4741737Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4741980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4742049Z return func(*args, **kwargs) 2025-11-03T16:36:17.4742314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4742389Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4742623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4742686Z return func(*args, **kwargs) 2025-11-03T16:36:17.4742960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4743081Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4743355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4743451Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4743455Z 2025-11-03T16:36:17.4743549Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4743738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4743799Z res = mod(**inputs) 2025-11-03T16:36:17.4744076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4744138Z outputs = self.bert( 2025-11-03T16:36:17.4744401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4744491Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4744760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4744835Z layer_outputs = layer_module( 2025-11-03T16:36:17.4745044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4745124Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4745349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4745414Z return func(*args, **kwargs) 2025-11-03T16:36:17.4745691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4745770Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4746020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4746090Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4746438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4746544Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4746813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4746897Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4746915Z 2025-11-03T16:36:17.4747011Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4747202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4747264Z res = mod(**inputs) 2025-11-03T16:36:17.4747534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4747606Z outputs = self.bert( 2025-11-03T16:36:17.4747880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4747956Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4748229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4748296Z layer_outputs = layer_module( 2025-11-03T16:36:17.4748518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4748591Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4748828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4748893Z return func(*args, **kwargs) 2025-11-03T16:36:17.4749175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4749272Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4749521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4749602Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4749905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4750013Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4750289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4750418Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4750633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4750702Z return self.act(input) 2025-11-03T16:36:17.4750707Z 2025-11-03T16:36:17.4750816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4751006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4751075Z res = mod(**inputs) 2025-11-03T16:36:17.4751355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4751421Z outputs = self.bert( 2025-11-03T16:36:17.4751703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4751774Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4752066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4752135Z layer_outputs = layer_module( 2025-11-03T16:36:17.4752362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4752446Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4752677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4752751Z return func(*args, **kwargs) 2025-11-03T16:36:17.4753050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4753140Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4753392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4753465Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4753783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4753909Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4754258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4754343Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4754347Z 2025-11-03T16:36:17.4754449Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4754648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4757155Z res = mod(**inputs) 2025-11-03T16:36:17.4757465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4757532Z outputs = self.bert( 2025-11-03T16:36:17.4757820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4757900Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4758178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4758255Z layer_outputs = layer_module( 2025-11-03T16:36:17.4758472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4758549Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4758792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4758912Z return func(*args, **kwargs) 2025-11-03T16:36:17.4759196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4759276Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4759508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4759582Z return func(*args, **kwargs) 2025-11-03T16:36:17.4759857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4759931Z self_outputs = self.self( 2025-11-03T16:36:17.4760167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4760234Z return func(*args, **kwargs) 2025-11-03T16:36:17.4760516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4760595Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4760598Z 2025-11-03T16:36:17.4760706Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4760922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4760993Z res = mod(**inputs) 2025-11-03T16:36:17.4761273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4761337Z outputs = self.bert( 2025-11-03T16:36:17.4761631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4761701Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4761986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4762053Z layer_outputs = layer_module( 2025-11-03T16:36:17.4762270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4762354Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4762582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4762654Z return func(*args, **kwargs) 2025-11-03T16:36:17.4762927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4763011Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4763240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4763373Z return func(*args, **kwargs) 2025-11-03T16:36:17.4763653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4763719Z self_outputs = self.self( 2025-11-03T16:36:17.4763957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4764023Z return func(*args, **kwargs) 2025-11-03T16:36:17.4764299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4764380Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4764384Z 2025-11-03T16:36:17.4764483Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4764680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4764758Z res = mod(**inputs) 2025-11-03T16:36:17.4765035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4765103Z outputs = self.bert( 2025-11-03T16:36:17.4765382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4765458Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4765735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4765810Z layer_outputs = layer_module( 2025-11-03T16:36:17.4766023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4766098Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4766337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4766402Z return func(*args, **kwargs) 2025-11-03T16:36:17.4766680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4766771Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4767001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4767073Z return func(*args, **kwargs) 2025-11-03T16:36:17.4767349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4767437Z self_outputs = self.self( 2025-11-03T16:36:17.4767669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4767747Z return func(*args, **kwargs) 2025-11-03T16:36:17.4768020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4768098Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4768101Z 2025-11-03T16:36:17.4768188Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4768265Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4768372Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4768559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4768620Z res = mod(**inputs) 2025-11-03T16:36:17.4768904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4768967Z outputs = self.bert( 2025-11-03T16:36:17.4769244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4769350Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4769622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4769701Z layer_outputs = layer_module( 2025-11-03T16:36:17.4769916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4769998Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4770232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4770307Z return func(*args, **kwargs) 2025-11-03T16:36:17.4770579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4770677Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4770914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4770978Z return func(*args, **kwargs) 2025-11-03T16:36:17.4771264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4771386Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4771656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4771743Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4771747Z 2025-11-03T16:36:17.4771844Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4772039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4772103Z res = mod(**inputs) 2025-11-03T16:36:17.4772388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4772451Z outputs = self.bert( 2025-11-03T16:36:17.4772736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4772817Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4773088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4773164Z layer_outputs = layer_module( 2025-11-03T16:36:17.4773389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4773464Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4773704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4773771Z return func(*args, **kwargs) 2025-11-03T16:36:17.4774051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4774133Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4774385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4774466Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4774767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4774875Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4775152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4775258Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4775261Z 2025-11-03T16:36:17.4775361Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4775553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4775625Z res = mod(**inputs) 2025-11-03T16:36:17.4775908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4775979Z outputs = self.bert( 2025-11-03T16:36:17.4776255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4776334Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4776610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4776696Z layer_outputs = layer_module( 2025-11-03T16:36:17.4776917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4776992Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4777231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4777294Z return func(*args, **kwargs) 2025-11-03T16:36:17.4777569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4777655Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4777908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4777986Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4778290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4778388Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4778685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4778796Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4779005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4779073Z return self.act(input) 2025-11-03T16:36:17.4779076Z 2025-11-03T16:36:17.4779178Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4779383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4779446Z res = mod(**inputs) 2025-11-03T16:36:17.4779736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4779798Z outputs = self.bert( 2025-11-03T16:36:17.4780081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4780150Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4780426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4780500Z layer_outputs = layer_module( 2025-11-03T16:36:17.4780713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4780797Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4781028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4782061Z return func(*args, **kwargs) 2025-11-03T16:36:17.4782337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4782417Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4782672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4782744Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4783081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4783214Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4783495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4783625Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4783628Z 2025-11-03T16:36:17.4783729Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4783931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4783996Z res = mod(**inputs) 2025-11-03T16:36:17.4784283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4784349Z outputs = self.bert( 2025-11-03T16:36:17.4784626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4784706Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4784984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4785062Z layer_outputs = layer_module( 2025-11-03T16:36:17.4785277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4785352Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4785606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4785675Z return func(*args, **kwargs) 2025-11-03T16:36:17.4785956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4786033Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4786301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4786378Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4786686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4786821Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4787095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4787176Z return input_tensor + hidden_states 2025-11-03T16:36:17.4787180Z 2025-11-03T16:36:17.4787278Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4787471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4787532Z res = mod(**inputs) 2025-11-03T16:36:17.4787811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4787882Z outputs = self.bert( 2025-11-03T16:36:17.4788160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4788253Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4788526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4788592Z layer_outputs = layer_module( 2025-11-03T16:36:17.4788809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4788883Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4789124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4789190Z return func(*args, **kwargs) 2025-11-03T16:36:17.4789465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4789568Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4789795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4789870Z return func(*args, **kwargs) 2025-11-03T16:36:17.4790139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4790212Z self_outputs = self.self( 2025-11-03T16:36:17.4790437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4790500Z return func(*args, **kwargs) 2025-11-03T16:36:17.4790774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4790850Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4790856Z 2025-11-03T16:36:17.4790960Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4791142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4791202Z res = mod(**inputs) 2025-11-03T16:36:17.4791508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4791570Z outputs = self.bert( 2025-11-03T16:36:17.4791848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4791917Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4792211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4792280Z layer_outputs = layer_module( 2025-11-03T16:36:17.4792493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4792577Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4792806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4792878Z return func(*args, **kwargs) 2025-11-03T16:36:17.4793159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4793238Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4793476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4793541Z return func(*args, **kwargs) 2025-11-03T16:36:17.4793821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4793906Z self_outputs = self.self( 2025-11-03T16:36:17.4794223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4794304Z return func(*args, **kwargs) 2025-11-03T16:36:17.4794587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4794668Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4794672Z 2025-11-03T16:36:17.4794772Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4794971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4795033Z res = mod(**inputs) 2025-11-03T16:36:17.4795314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4795410Z outputs = self.bert( 2025-11-03T16:36:17.4795688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4795767Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4796049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4796119Z layer_outputs = layer_module( 2025-11-03T16:36:17.4796340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4796415Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4796657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4796722Z return func(*args, **kwargs) 2025-11-03T16:36:17.4797015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4797099Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4797333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4797409Z return func(*args, **kwargs) 2025-11-03T16:36:17.4797700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4797776Z self_outputs = self.self( 2025-11-03T16:36:17.4798009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4798073Z return func(*args, **kwargs) 2025-11-03T16:36:17.4798377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4798454Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4798460Z 2025-11-03T16:36:17.4798546Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4798622Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4798719Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4798915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4798978Z res = mod(**inputs) 2025-11-03T16:36:17.4799268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4799331Z outputs = self.bert( 2025-11-03T16:36:17.4799616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4799686Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4799962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4800056Z layer_outputs = layer_module( 2025-11-03T16:36:17.4800273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4800355Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4800594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4800659Z return func(*args, **kwargs) 2025-11-03T16:36:17.4800953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4801039Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4801287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4801353Z return func(*args, **kwargs) 2025-11-03T16:36:17.4801646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4801777Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4802053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4802139Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4802142Z 2025-11-03T16:36:17.4802240Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4802436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4802497Z res = mod(**inputs) 2025-11-03T16:36:17.4802776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4802849Z outputs = self.bert( 2025-11-03T16:36:17.4803125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4803203Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4803497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4803572Z layer_outputs = layer_module( 2025-11-03T16:36:17.4803779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4803851Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4804099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4804163Z return func(*args, **kwargs) 2025-11-03T16:36:17.4804439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4804519Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4804759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4804840Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4805140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4805246Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4805512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4805596Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4805599Z 2025-11-03T16:36:17.4805694Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4805880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4805979Z res = mod(**inputs) 2025-11-03T16:36:17.4806248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4806315Z outputs = self.bert( 2025-11-03T16:36:17.4806587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4806658Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4806930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4806998Z layer_outputs = layer_module( 2025-11-03T16:36:17.4807212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4807303Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4807527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4807597Z return func(*args, **kwargs) 2025-11-03T16:36:17.4807864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4807949Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4808189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4808268Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4808559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4808655Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4808933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4809039Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4809244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4809328Z return self.act(input) 2025-11-03T16:36:17.4809332Z 2025-11-03T16:36:17.4809434Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4809619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4809679Z res = mod(**inputs) 2025-11-03T16:36:17.4809982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4810045Z outputs = self.bert( 2025-11-03T16:36:17.4810315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4810386Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4810650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4810724Z layer_outputs = layer_module( 2025-11-03T16:36:17.4810928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4811008Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4811229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4811294Z return func(*args, **kwargs) 2025-11-03T16:36:17.4811568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4811663Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4811909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4811979Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4812283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4812406Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4812672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4812757Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4812763Z 2025-11-03T16:36:17.4812860Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4813048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4813131Z res = mod(**inputs) 2025-11-03T16:36:17.4813572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4813649Z outputs = self.bert( 2025-11-03T16:36:17.4813919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4813996Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4814263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4814340Z layer_outputs = layer_module( 2025-11-03T16:36:17.4814549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4814624Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4814861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4814925Z return func(*args, **kwargs) 2025-11-03T16:36:17.4815201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4815325Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4815551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4815623Z return func(*args, **kwargs) 2025-11-03T16:36:17.4815887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4815982Z self_outputs = self.self( 2025-11-03T16:36:17.4816208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4816283Z return func(*args, **kwargs) 2025-11-03T16:36:17.4816549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4816627Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4816631Z 2025-11-03T16:36:17.4816738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4816923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4816992Z res = mod(**inputs) 2025-11-03T16:36:17.4817261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4817325Z outputs = self.bert( 2025-11-03T16:36:17.4817601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4817720Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4817991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4818057Z layer_outputs = layer_module( 2025-11-03T16:36:17.4818263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4818344Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4818567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4818640Z return func(*args, **kwargs) 2025-11-03T16:36:17.4818907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4818990Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4819242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4819309Z return func(*args, **kwargs) 2025-11-03T16:36:17.4819585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4819653Z self_outputs = self.self( 2025-11-03T16:36:17.4819885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4819950Z return func(*args, **kwargs) 2025-11-03T16:36:17.4820217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4820301Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4820305Z 2025-11-03T16:36:17.4820401Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4820596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4820656Z res = mod(**inputs) 2025-11-03T16:36:17.4820938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4820998Z outputs = self.bert( 2025-11-03T16:36:17.4821286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4821363Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4821629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4821703Z layer_outputs = layer_module( 2025-11-03T16:36:17.4821925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4822003Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4822238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4822300Z return func(*args, **kwargs) 2025-11-03T16:36:17.4822574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4822652Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4822882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4822948Z return func(*args, **kwargs) 2025-11-03T16:36:17.4823214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4823285Z self_outputs = self.self( 2025-11-03T16:36:17.4823512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4823603Z return func(*args, **kwargs) 2025-11-03T16:36:17.4823876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4823953Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4823958Z 2025-11-03T16:36:17.4824039Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4824113Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4824214Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4824395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4824454Z res = mod(**inputs) 2025-11-03T16:36:17.4824732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4824793Z outputs = self.bert( 2025-11-03T16:36:17.4825090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4825158Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4825437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4825503Z layer_outputs = layer_module( 2025-11-03T16:36:17.4825712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4825790Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4826016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4826087Z return func(*args, **kwargs) 2025-11-03T16:36:17.4826356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4826435Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4826667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4826730Z return func(*args, **kwargs) 2025-11-03T16:36:17.4827025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4827148Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4827422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4827526Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4827530Z 2025-11-03T16:36:17.4827628Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4827819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4827882Z res = mod(**inputs) 2025-11-03T16:36:17.4828158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4828220Z outputs = self.bert( 2025-11-03T16:36:17.4828491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4828567Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4828831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4828904Z layer_outputs = layer_module( 2025-11-03T16:36:17.4829112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4829185Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4829434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4829498Z return func(*args, **kwargs) 2025-11-03T16:36:17.4829769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4829849Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4830101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4830173Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4830467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4830574Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4830857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4830941Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4830944Z 2025-11-03T16:36:17.4831040Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4831231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4831292Z res = mod(**inputs) 2025-11-03T16:36:17.4831563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4831631Z outputs = self.bert( 2025-11-03T16:36:17.4831898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4831973Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4832238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4832305Z layer_outputs = layer_module( 2025-11-03T16:36:17.4832517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4832606Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4832839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4832903Z return func(*args, **kwargs) 2025-11-03T16:36:17.4833172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4833272Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4833514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4833595Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4833890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4833997Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4834334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4834448Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4834661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4834728Z return self.act(input) 2025-11-03T16:36:17.4834731Z 2025-11-03T16:36:17.4834840Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4835032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4835119Z res = mod(**inputs) 2025-11-03T16:36:17.4835406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4835470Z outputs = self.bert( 2025-11-03T16:36:17.4835766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4835835Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4836136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4836205Z layer_outputs = layer_module( 2025-11-03T16:36:17.4836418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4836504Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4836755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4836831Z return func(*args, **kwargs) 2025-11-03T16:36:17.4837116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4837200Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4837457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4837529Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4837838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4837965Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4838251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4838330Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4838333Z 2025-11-03T16:36:17.4838432Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4838629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4838706Z res = mod(**inputs) 2025-11-03T16:36:17.4838991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4839055Z outputs = self.bert( 2025-11-03T16:36:17.4839344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4839421Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4839698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4839775Z layer_outputs = layer_module( 2025-11-03T16:36:17.4839989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4840069Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4840305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4840370Z return func(*args, **kwargs) 2025-11-03T16:36:17.4840654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4840731Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4840986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4841058Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4841378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4841512Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4841788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4841871Z return input_tensor + hidden_states 2025-11-03T16:36:17.4841874Z 2025-11-03T16:36:17.4841970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4842164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4842226Z res = mod(**inputs) 2025-11-03T16:36:17.4842520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4842652Z outputs = self.bert( 2025-11-03T16:36:17.4842929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4843007Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4843284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4843351Z layer_outputs = layer_module( 2025-11-03T16:36:17.4843572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4843646Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4843889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4843956Z return func(*args, **kwargs) 2025-11-03T16:36:17.4844241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4844322Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4844558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4844631Z return func(*args, **kwargs) 2025-11-03T16:36:17.4844923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4844999Z self_outputs = self.self( 2025-11-03T16:36:17.4845231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4845295Z return func(*args, **kwargs) 2025-11-03T16:36:17.4845591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4845670Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4845675Z 2025-11-03T16:36:17.4845778Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4845967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4846035Z res = mod(**inputs) 2025-11-03T16:36:17.4846312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4846375Z outputs = self.bert( 2025-11-03T16:36:17.4846651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4846719Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4847000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4847067Z layer_outputs = layer_module( 2025-11-03T16:36:17.4847297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4847381Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4847611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4847685Z return func(*args, **kwargs) 2025-11-03T16:36:17.4847962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4848040Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4848286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4848349Z return func(*args, **kwargs) 2025-11-03T16:36:17.4848621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4848702Z self_outputs = self.self( 2025-11-03T16:36:17.4848932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4848995Z return func(*args, **kwargs) 2025-11-03T16:36:17.4849261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4849341Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4849344Z 2025-11-03T16:36:17.4849440Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4849627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4849687Z res = mod(**inputs) 2025-11-03T16:36:17.4849955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4850027Z outputs = self.bert( 2025-11-03T16:36:17.4850290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4850366Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4850666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4850740Z layer_outputs = layer_module( 2025-11-03T16:36:17.4850949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4851020Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4851267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4851331Z return func(*args, **kwargs) 2025-11-03T16:36:17.4851604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4851681Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4851906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4851976Z return func(*args, **kwargs) 2025-11-03T16:36:17.4852241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4852312Z self_outputs = self.self( 2025-11-03T16:36:17.4852537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4852601Z return func(*args, **kwargs) 2025-11-03T16:36:17.4852870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4852960Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4852963Z 2025-11-03T16:36:17.4853044Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4853118Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4853222Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4853406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4853464Z res = mod(**inputs) 2025-11-03T16:36:17.4853742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4853803Z outputs = self.bert( 2025-11-03T16:36:17.4854074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4854142Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4854423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4854500Z layer_outputs = layer_module( 2025-11-03T16:36:17.4854704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4854786Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4855011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4855077Z return func(*args, **kwargs) 2025-11-03T16:36:17.4855355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4855433Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4855668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4855734Z return func(*args, **kwargs) 2025-11-03T16:36:17.4856015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4856137Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4856439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4856528Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4856532Z 2025-11-03T16:36:17.4856629Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4856825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4856903Z res = mod(**inputs) 2025-11-03T16:36:17.4857182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4857257Z outputs = self.bert( 2025-11-03T16:36:17.4857532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4857610Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4857887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4857963Z layer_outputs = layer_module( 2025-11-03T16:36:17.4858176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4858250Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4858489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4858555Z return func(*args, **kwargs) 2025-11-03T16:36:17.4858843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4858966Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4859214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4859294Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4859595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4859708Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4859985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4860073Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4860076Z 2025-11-03T16:36:17.4860191Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4860381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4860451Z res = mod(**inputs) 2025-11-03T16:36:17.4860729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4860801Z outputs = self.bert( 2025-11-03T16:36:17.4861075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4861144Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4861428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4861494Z layer_outputs = layer_module( 2025-11-03T16:36:17.4861715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4861791Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4862031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4862097Z return func(*args, **kwargs) 2025-11-03T16:36:17.4862387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4862477Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4862736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4862815Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4863136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4863238Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4863531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4863639Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4863848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4863914Z return self.act(input) 2025-11-03T16:36:17.4863917Z 2025-11-03T16:36:17.4864019Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4864208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4864268Z res = mod(**inputs) 2025-11-03T16:36:17.4864548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4864609Z outputs = self.bert( 2025-11-03T16:36:17.4864899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4864966Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4865231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4865308Z layer_outputs = layer_module( 2025-11-03T16:36:17.4865512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4865592Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4865814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4865886Z return func(*args, **kwargs) 2025-11-03T16:36:17.4866150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4866246Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4866494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4866563Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4866863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4866988Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4867254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4867339Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4867342Z 2025-11-03T16:36:17.4867437Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4867629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4867691Z res = mod(**inputs) 2025-11-03T16:36:17.4867967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4868029Z outputs = self.bert( 2025-11-03T16:36:17.4868311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4868389Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4868663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4868748Z layer_outputs = layer_module( 2025-11-03T16:36:17.4868956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4869031Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4869266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4869329Z return func(*args, **kwargs) 2025-11-03T16:36:17.4869605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4869681Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4869911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4869974Z return func(*args, **kwargs) 2025-11-03T16:36:17.4870241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4870313Z self_outputs = self.self( 2025-11-03T16:36:17.4870540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4870626Z return func(*args, **kwargs) 2025-11-03T16:36:17.4870896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4870974Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4870977Z 2025-11-03T16:36:17.4871079Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4871261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4871329Z res = mod(**inputs) 2025-11-03T16:36:17.4871601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4871662Z outputs = self.bert( 2025-11-03T16:36:17.4871932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4872023Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4872296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4872361Z layer_outputs = layer_module( 2025-11-03T16:36:17.4872576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4872647Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4872871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4872947Z return func(*args, **kwargs) 2025-11-03T16:36:17.4873213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4873296Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4873524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4873587Z return func(*args, **kwargs) 2025-11-03T16:36:17.4873886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4873955Z self_outputs = self.self( 2025-11-03T16:36:17.4874270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4874341Z return func(*args, **kwargs) 2025-11-03T16:36:17.4874644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4874722Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4874726Z 2025-11-03T16:36:17.4874825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4875030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4875093Z res = mod(**inputs) 2025-11-03T16:36:17.4875391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4875458Z outputs = self.bert( 2025-11-03T16:36:17.4875734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4875815Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4876090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4876169Z layer_outputs = layer_module( 2025-11-03T16:36:17.4876383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4876487Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4876717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4876784Z return func(*args, **kwargs) 2025-11-03T16:36:17.4877072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4877153Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4877396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4877463Z return func(*args, **kwargs) 2025-11-03T16:36:17.4877745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4877822Z self_outputs = self.self( 2025-11-03T16:36:17.4878072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4878149Z return func(*args, **kwargs) 2025-11-03T16:36:17.4878424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4878501Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4878511Z 2025-11-03T16:36:17.4878589Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4878664Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4878767Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4878956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4879027Z res = mod(**inputs) 2025-11-03T16:36:17.4879305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4879369Z outputs = self.bert( 2025-11-03T16:36:17.4879650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4879718Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4880015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4880083Z layer_outputs = layer_module( 2025-11-03T16:36:17.4880295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4880375Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4880620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4880693Z return func(*args, **kwargs) 2025-11-03T16:36:17.4880974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4881053Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4881289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4881354Z return func(*args, **kwargs) 2025-11-03T16:36:17.4881634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4881755Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4882040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4882119Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4882122Z 2025-11-03T16:36:17.4882220Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4882434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4882495Z res = mod(**inputs) 2025-11-03T16:36:17.4882788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4882860Z outputs = self.bert( 2025-11-03T16:36:17.4883140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4883217Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4883494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4883574Z layer_outputs = layer_module( 2025-11-03T16:36:17.4883785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4883889Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4884121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4884186Z return func(*args, **kwargs) 2025-11-03T16:36:17.4884468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4884547Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4884804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4884878Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4885189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4885297Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4885579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4885666Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4885670Z 2025-11-03T16:36:17.4885768Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4885982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4886045Z res = mod(**inputs) 2025-11-03T16:36:17.4886320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4886391Z outputs = self.bert( 2025-11-03T16:36:17.4886677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4886754Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4887039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4887106Z layer_outputs = layer_module( 2025-11-03T16:36:17.4887331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4887406Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4887650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4887714Z return func(*args, **kwargs) 2025-11-03T16:36:17.4888005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4888085Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4888339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4888439Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4888738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4888847Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4889117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4889226Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4889434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4889503Z return self.act(input) 2025-11-03T16:36:17.4889506Z 2025-11-03T16:36:17.4889610Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4889794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4889884Z res = mod(**inputs) 2025-11-03T16:36:17.4890167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4890232Z outputs = self.bert( 2025-11-03T16:36:17.4890520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4890589Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4890875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4890943Z layer_outputs = layer_module( 2025-11-03T16:36:17.4891161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4891243Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4891482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4891555Z return func(*args, **kwargs) 2025-11-03T16:36:17.4891855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4891944Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4892190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4892261Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4892582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4892709Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4892989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4893070Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4893073Z 2025-11-03T16:36:17.4893168Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4893369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4893432Z res = mod(**inputs) 2025-11-03T16:36:17.4893721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4893795Z outputs = self.bert( 2025-11-03T16:36:17.4894074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4894141Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4894413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4894508Z layer_outputs = layer_module( 2025-11-03T16:36:17.4894714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4894795Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4895020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4895083Z return func(*args, **kwargs) 2025-11-03T16:36:17.4895362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4895443Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4895691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4895778Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4896079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4896202Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4896469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4896550Z return input_tensor + hidden_states 2025-11-03T16:36:17.4896554Z 2025-11-03T16:36:17.4896648Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4896837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4896897Z res = mod(**inputs) 2025-11-03T16:36:17.4897163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4897237Z outputs = self.bert( 2025-11-03T16:36:17.4897508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4897584Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4897881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4897958Z layer_outputs = layer_module( 2025-11-03T16:36:17.4898168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4898243Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4898497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4898565Z return func(*args, **kwargs) 2025-11-03T16:36:17.4898853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4898933Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4899168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4899243Z return func(*args, **kwargs) 2025-11-03T16:36:17.4899523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4899597Z self_outputs = self.self( 2025-11-03T16:36:17.4899830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4899895Z return func(*args, **kwargs) 2025-11-03T16:36:17.4900177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4900278Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4900281Z 2025-11-03T16:36:17.4900387Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4900577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4900648Z res = mod(**inputs) 2025-11-03T16:36:17.4900928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4900990Z outputs = self.bert( 2025-11-03T16:36:17.4901271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4901341Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4901622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4901709Z layer_outputs = layer_module( 2025-11-03T16:36:17.4901920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4902000Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4902237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4902311Z return func(*args, **kwargs) 2025-11-03T16:36:17.4902587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4902673Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4902908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4902975Z return func(*args, **kwargs) 2025-11-03T16:36:17.4903253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4903326Z self_outputs = self.self( 2025-11-03T16:36:17.4903564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4903630Z return func(*args, **kwargs) 2025-11-03T16:36:17.4903918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4904002Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4904006Z 2025-11-03T16:36:17.4904102Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4904317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4904380Z res = mod(**inputs) 2025-11-03T16:36:17.4904667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4904733Z outputs = self.bert( 2025-11-03T16:36:17.4905007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4905085Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4905358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4905432Z layer_outputs = layer_module( 2025-11-03T16:36:17.4905642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4905715Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4905955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4906020Z return func(*args, **kwargs) 2025-11-03T16:36:17.4906311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4906388Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4906619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4906690Z return func(*args, **kwargs) 2025-11-03T16:36:17.4906962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4907038Z self_outputs = self.self( 2025-11-03T16:36:17.4907270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4907337Z return func(*args, **kwargs) 2025-11-03T16:36:17.4907619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4907712Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4907716Z 2025-11-03T16:36:17.4907799Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4907875Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4907981Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4908170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4908228Z res = mod(**inputs) 2025-11-03T16:36:17.4908511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4908573Z outputs = self.bert( 2025-11-03T16:36:17.4908853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4908922Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4909195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4909266Z layer_outputs = layer_module( 2025-11-03T16:36:17.4909474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4909569Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4909800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4909871Z return func(*args, **kwargs) 2025-11-03T16:36:17.4910163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4910241Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4910480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4910547Z return func(*args, **kwargs) 2025-11-03T16:36:17.4910824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4910948Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4911227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4911309Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4911312Z 2025-11-03T16:36:17.4911409Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4911606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4911666Z res = mod(**inputs) 2025-11-03T16:36:17.4911951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4912034Z outputs = self.bert( 2025-11-03T16:36:17.4912302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4912380Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4912657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4912732Z layer_outputs = layer_module( 2025-11-03T16:36:17.4912941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4913012Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4913419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4913537Z return func(*args, **kwargs) 2025-11-03T16:36:17.4913829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4913912Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4914216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4914297Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4914608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4914721Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4915006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4915095Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4915101Z 2025-11-03T16:36:17.4915204Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4915403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4915476Z res = mod(**inputs) 2025-11-03T16:36:17.4915793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4915867Z outputs = self.bert( 2025-11-03T16:36:17.4916155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4916233Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4916547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4916618Z layer_outputs = layer_module( 2025-11-03T16:36:17.4916841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4916915Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4917156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4917221Z return func(*args, **kwargs) 2025-11-03T16:36:17.4917499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4917588Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4917840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4917925Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4918229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4918365Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4918638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4918750Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4918970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4919038Z return self.act(input) 2025-11-03T16:36:17.4919041Z 2025-11-03T16:36:17.4919150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4919341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4919404Z res = mod(**inputs) 2025-11-03T16:36:17.4919687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4919768Z outputs = self.bert( 2025-11-03T16:36:17.4920044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4920112Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4920388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4920453Z layer_outputs = layer_module( 2025-11-03T16:36:17.4920661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4920738Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4920969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4921037Z return func(*args, **kwargs) 2025-11-03T16:36:17.4921308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4921388Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4921637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4921720Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4922023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4922146Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4922433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4922510Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4922513Z 2025-11-03T16:36:17.4922610Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4922802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4922861Z res = mod(**inputs) 2025-11-03T16:36:17.4923137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4923199Z outputs = self.bert( 2025-11-03T16:36:17.4923466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4923539Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4923811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4923881Z layer_outputs = layer_module( 2025-11-03T16:36:17.4924088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4924183Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4924410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4924472Z return func(*args, **kwargs) 2025-11-03T16:36:17.4924749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4924824Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4925058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4925121Z return func(*args, **kwargs) 2025-11-03T16:36:17.4925392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4925462Z self_outputs = self.self( 2025-11-03T16:36:17.4925704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4925771Z return func(*args, **kwargs) 2025-11-03T16:36:17.4926040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4926116Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4926124Z 2025-11-03T16:36:17.4926217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4926402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4926466Z res = mod(**inputs) 2025-11-03T16:36:17.4926738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4926803Z outputs = self.bert( 2025-11-03T16:36:17.4927071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4927139Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4927411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4927488Z layer_outputs = layer_module( 2025-11-03T16:36:17.4927705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4927777Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4928005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4928085Z return func(*args, **kwargs) 2025-11-03T16:36:17.4928357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4928438Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4928666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4928733Z return func(*args, **kwargs) 2025-11-03T16:36:17.4929004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4929069Z self_outputs = self.self( 2025-11-03T16:36:17.4929300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4929362Z return func(*args, **kwargs) 2025-11-03T16:36:17.4929635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4929708Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4929713Z 2025-11-03T16:36:17.4929823Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4930012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4930071Z res = mod(**inputs) 2025-11-03T16:36:17.4930354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4930413Z outputs = self.bert( 2025-11-03T16:36:17.4930681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4930753Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4931028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4931098Z layer_outputs = layer_module( 2025-11-03T16:36:17.4931304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4931396Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4931623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4931686Z return func(*args, **kwargs) 2025-11-03T16:36:17.4931963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4932038Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4932268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4932330Z return func(*args, **kwargs) 2025-11-03T16:36:17.4932606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4932678Z self_outputs = self.self( 2025-11-03T16:36:17.4932905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4932972Z return func(*args, **kwargs) 2025-11-03T16:36:17.4933262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4933344Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4933347Z 2025-11-03T16:36:17.4933423Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4933497Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4933597Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4933796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4933861Z res = mod(**inputs) 2025-11-03T16:36:17.4934137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4934200Z outputs = self.bert( 2025-11-03T16:36:17.4934476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4934543Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4934824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4934890Z layer_outputs = layer_module( 2025-11-03T16:36:17.4935098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4935175Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4935404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4935472Z return func(*args, **kwargs) 2025-11-03T16:36:17.4935793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4935873Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4936103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4936165Z return func(*args, **kwargs) 2025-11-03T16:36:17.4936449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4936569Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4936850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4936926Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4936947Z 2025-11-03T16:36:17.4937044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4937241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4937300Z res = mod(**inputs) 2025-11-03T16:36:17.4937583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4937644Z outputs = self.bert( 2025-11-03T16:36:17.4937925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4937994Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4938271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4938342Z layer_outputs = layer_module( 2025-11-03T16:36:17.4938552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4938630Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4938859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4938921Z return func(*args, **kwargs) 2025-11-03T16:36:17.4939224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4939303Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4939553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4939623Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4939946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4940048Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4940323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4940407Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4940410Z 2025-11-03T16:36:17.4940506Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4940700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4940759Z res = mod(**inputs) 2025-11-03T16:36:17.4941033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4941101Z outputs = self.bert( 2025-11-03T16:36:17.4941373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4941463Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4941736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4941805Z layer_outputs = layer_module( 2025-11-03T16:36:17.4942017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4942091Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4942325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4942388Z return func(*args, **kwargs) 2025-11-03T16:36:17.4942664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4942741Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4943001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4943079Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4943378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4943480Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4943752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4943862Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4944067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4944132Z return self.act(input) 2025-11-03T16:36:17.4944136Z 2025-11-03T16:36:17.4944236Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4944423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4944488Z res = mod(**inputs) 2025-11-03T16:36:17.4944763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4944844Z outputs = self.bert( 2025-11-03T16:36:17.4945126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4945193Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4945468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4945545Z layer_outputs = layer_module( 2025-11-03T16:36:17.4945761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4945837Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4946067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4946136Z return func(*args, **kwargs) 2025-11-03T16:36:17.4946416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4946498Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4946744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4946813Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4947123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4947246Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4947542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4947616Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4947620Z 2025-11-03T16:36:17.4947721Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4947915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4947974Z res = mod(**inputs) 2025-11-03T16:36:17.4948260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4948322Z outputs = self.bert( 2025-11-03T16:36:17.4948612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4948679Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4948968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4949039Z layer_outputs = layer_module( 2025-11-03T16:36:17.4949248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4949327Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4949554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4949617Z return func(*args, **kwargs) 2025-11-03T16:36:17.4949896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4949972Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4950220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4950292Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4950593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4950727Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4950996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.4951072Z return input_tensor + hidden_states 2025-11-03T16:36:17.4951075Z 2025-11-03T16:36:17.4951170Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4951379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4951440Z res = mod(**inputs) 2025-11-03T16:36:17.4951724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4951792Z outputs = self.bert( 2025-11-03T16:36:17.4952064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4952141Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4952419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4952493Z layer_outputs = layer_module( 2025-11-03T16:36:17.4952702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4952778Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4953016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4953084Z return func(*args, **kwargs) 2025-11-03T16:36:17.4953387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4953465Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4953694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4953767Z return func(*args, **kwargs) 2025-11-03T16:36:17.4954100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4954183Z self_outputs = self.self( 2025-11-03T16:36:17.4954413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4954489Z return func(*args, **kwargs) 2025-11-03T16:36:17.4954765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4954867Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4954871Z 2025-11-03T16:36:17.4954980Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4955171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4955240Z res = mod(**inputs) 2025-11-03T16:36:17.4955519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4955582Z outputs = self.bert( 2025-11-03T16:36:17.4955870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4955943Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4956226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4956298Z layer_outputs = layer_module( 2025-11-03T16:36:17.4956523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4956599Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4956850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4956923Z return func(*args, **kwargs) 2025-11-03T16:36:17.4957204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4957289Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4957536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4957603Z return func(*args, **kwargs) 2025-11-03T16:36:17.4957899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4957965Z self_outputs = self.self( 2025-11-03T16:36:17.4958198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4958263Z return func(*args, **kwargs) 2025-11-03T16:36:17.4958535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4958615Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4958618Z 2025-11-03T16:36:17.4958718Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4958914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4958979Z res = mod(**inputs) 2025-11-03T16:36:17.4959264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4959347Z outputs = self.bert( 2025-11-03T16:36:17.4959620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4959697Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4959975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4960048Z layer_outputs = layer_module( 2025-11-03T16:36:17.4960261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4960337Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4960574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4960662Z return func(*args, **kwargs) 2025-11-03T16:36:17.4960941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4961018Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4961257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4961321Z return func(*args, **kwargs) 2025-11-03T16:36:17.4961602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4961675Z self_outputs = self.self( 2025-11-03T16:36:17.4961907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4961975Z return func(*args, **kwargs) 2025-11-03T16:36:17.4962248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4962327Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4962330Z 2025-11-03T16:36:17.4962411Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4962486Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4962606Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4962794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4962856Z res = mod(**inputs) 2025-11-03T16:36:17.4963140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4963218Z outputs = self.bert( 2025-11-03T16:36:17.4963496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4963565Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4963840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4963906Z layer_outputs = layer_module( 2025-11-03T16:36:17.4964120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4964201Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4964427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4964494Z return func(*args, **kwargs) 2025-11-03T16:36:17.4964766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4964841Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4965074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4965154Z return func(*args, **kwargs) 2025-11-03T16:36:17.4965433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4965557Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4965835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4965915Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4965918Z 2025-11-03T16:36:17.4966013Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4966210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4966268Z res = mod(**inputs) 2025-11-03T16:36:17.4966569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4966630Z outputs = self.bert( 2025-11-03T16:36:17.4966899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4966972Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4967243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4967312Z layer_outputs = layer_module( 2025-11-03T16:36:17.4967520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4967597Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4967823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4967889Z return func(*args, **kwargs) 2025-11-03T16:36:17.4968164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4968240Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4968504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4968575Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4968874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4968978Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4969262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4969346Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4969350Z 2025-11-03T16:36:17.4969444Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4969637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4969695Z res = mod(**inputs) 2025-11-03T16:36:17.4969972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4970038Z outputs = self.bert( 2025-11-03T16:36:17.4970310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4970383Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4970655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4970721Z layer_outputs = layer_module( 2025-11-03T16:36:17.4970952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4971023Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4971254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4971318Z return func(*args, **kwargs) 2025-11-03T16:36:17.4971593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4971669Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4971915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4971990Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4972286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4972407Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4972678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4972784Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4972993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4973058Z return self.act(input) 2025-11-03T16:36:17.4973061Z 2025-11-03T16:36:17.4973161Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4973350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4973416Z res = mod(**inputs) 2025-11-03T16:36:17.4973694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4973760Z outputs = self.bert( 2025-11-03T16:36:17.4974043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4974112Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4974409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4974479Z layer_outputs = layer_module( 2025-11-03T16:36:17.4974687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4974769Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4975021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4975096Z return func(*args, **kwargs) 2025-11-03T16:36:17.4975372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4975453Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4975708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4975779Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4976087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.4976214Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.4976500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.4976579Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4976597Z 2025-11-03T16:36:17.4976697Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4976893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4976954Z res = mod(**inputs) 2025-11-03T16:36:17.4977242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4977311Z outputs = self.bert( 2025-11-03T16:36:17.4977602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4977671Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4977940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4978017Z layer_outputs = layer_module( 2025-11-03T16:36:17.4978245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4978329Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4978559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4978624Z return func(*args, **kwargs) 2025-11-03T16:36:17.4978906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4978984Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4979219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4979285Z return func(*args, **kwargs) 2025-11-03T16:36:17.4979557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4979633Z self_outputs = self.self( 2025-11-03T16:36:17.4979865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4979936Z return func(*args, **kwargs) 2025-11-03T16:36:17.4980230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.4980317Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.4980320Z 2025-11-03T16:36:17.4980417Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4980606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4980675Z res = mod(**inputs) 2025-11-03T16:36:17.4980971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4981044Z outputs = self.bert( 2025-11-03T16:36:17.4981322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4981390Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4981681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4981747Z layer_outputs = layer_module( 2025-11-03T16:36:17.4981967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4982040Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4982280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4982345Z return func(*args, **kwargs) 2025-11-03T16:36:17.4982622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4982738Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4982974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4983045Z return func(*args, **kwargs) 2025-11-03T16:36:17.4983323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4983388Z self_outputs = self.self( 2025-11-03T16:36:17.4983629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4983692Z return func(*args, **kwargs) 2025-11-03T16:36:17.4983980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.4984073Z key_layer = self.key(current_states) 2025-11-03T16:36:17.4984078Z 2025-11-03T16:36:17.4984180Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4984366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4984426Z res = mod(**inputs) 2025-11-03T16:36:17.4984715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4984776Z outputs = self.bert( 2025-11-03T16:36:17.4985055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4985123Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4985399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4985474Z layer_outputs = layer_module( 2025-11-03T16:36:17.4985685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4985765Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4985995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4986075Z return func(*args, **kwargs) 2025-11-03T16:36:17.4986356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4986434Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4986668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4986747Z return func(*args, **kwargs) 2025-11-03T16:36:17.4987026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.4987095Z self_outputs = self.self( 2025-11-03T16:36:17.4987324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4987395Z return func(*args, **kwargs) 2025-11-03T16:36:17.4987670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.4987752Z value_layer = self.value(current_states) 2025-11-03T16:36:17.4987755Z 2025-11-03T16:36:17.4987832Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4987908Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.4988014Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4988206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4988272Z res = mod(**inputs) 2025-11-03T16:36:17.4988550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4988628Z outputs = self.bert( 2025-11-03T16:36:17.4988912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4988982Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4989264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4989330Z layer_outputs = layer_module( 2025-11-03T16:36:17.4989548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4989623Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4989850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4989941Z return func(*args, **kwargs) 2025-11-03T16:36:17.4990218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.4990304Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.4990536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4990601Z return func(*args, **kwargs) 2025-11-03T16:36:17.4990888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.4991011Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.4991298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.4991379Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4991384Z 2025-11-03T16:36:17.4991491Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4991679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4991743Z res = mod(**inputs) 2025-11-03T16:36:17.4992045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4992109Z outputs = self.bert( 2025-11-03T16:36:17.4992389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4992458Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4992746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4992822Z layer_outputs = layer_module( 2025-11-03T16:36:17.4993034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4993115Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4993343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4993416Z return func(*args, **kwargs) 2025-11-03T16:36:17.4993688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4993768Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4994087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4994181Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4994497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4994623Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4994905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.4994995Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.4994999Z 2025-11-03T16:36:17.4995100Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4995309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4995383Z res = mod(**inputs) 2025-11-03T16:36:17.4995668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4995732Z outputs = self.bert( 2025-11-03T16:36:17.4996009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.4996111Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.4996391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.4996469Z layer_outputs = layer_module( 2025-11-03T16:36:17.4996689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.4996766Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.4997017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.4997085Z return func(*args, **kwargs) 2025-11-03T16:36:17.4997393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.4997476Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.4997742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.4997824Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.4998157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.4998271Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.4998555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.4998675Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.4998901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.4998971Z return self.act(input) 2025-11-03T16:36:17.4998976Z 2025-11-03T16:36:17.4999085Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.4999283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.4999352Z res = mod(**inputs) 2025-11-03T16:36:17.4999638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.4999702Z outputs = self.bert( 2025-11-03T16:36:17.4999986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5000056Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5000350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5000418Z layer_outputs = layer_module( 2025-11-03T16:36:17.5000645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5000741Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5000995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5001070Z return func(*args, **kwargs) 2025-11-03T16:36:17.5001358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5001447Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5001708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5001783Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5002108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5002256Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5002548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5002628Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5002632Z 2025-11-03T16:36:17.5002739Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5002937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5003001Z res = mod(**inputs) 2025-11-03T16:36:17.5003303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5003368Z outputs = self.bert( 2025-11-03T16:36:17.5003664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5003738Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5004026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5004104Z layer_outputs = layer_module( 2025-11-03T16:36:17.5004341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5004428Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5004670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5004746Z return func(*args, **kwargs) 2025-11-03T16:36:17.5005048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5005127Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5005377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5005449Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5005752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5005878Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5006144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.5006224Z return input_tensor + hidden_states 2025-11-03T16:36:17.5006228Z 2025-11-03T16:36:17.5006321Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5006513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5006574Z res = mod(**inputs) 2025-11-03T16:36:17.5006870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5006930Z outputs = self.bert( 2025-11-03T16:36:17.5007197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5007273Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5007543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5007615Z layer_outputs = layer_module( 2025-11-03T16:36:17.5007823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5007898Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5008130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5008212Z return func(*args, **kwargs) 2025-11-03T16:36:17.5008489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5008565Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5008797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5008861Z return func(*args, **kwargs) 2025-11-03T16:36:17.5009127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5009202Z self_outputs = self.self( 2025-11-03T16:36:17.5009430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5009502Z return func(*args, **kwargs) 2025-11-03T16:36:17.5009771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5009847Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5009850Z 2025-11-03T16:36:17.5009951Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5010151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5010219Z res = mod(**inputs) 2025-11-03T16:36:17.5010492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5010559Z outputs = self.bert( 2025-11-03T16:36:17.5010857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5010925Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5011195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5011262Z layer_outputs = layer_module( 2025-11-03T16:36:17.5011475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5011550Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5011776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5011845Z return func(*args, **kwargs) 2025-11-03T16:36:17.5012109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5012191Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5012414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5012492Z return func(*args, **kwargs) 2025-11-03T16:36:17.5012766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5012829Z self_outputs = self.self( 2025-11-03T16:36:17.5013061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5013124Z return func(*args, **kwargs) 2025-11-03T16:36:17.5013512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5013591Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5013595Z 2025-11-03T16:36:17.5013695Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5013891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5013995Z res = mod(**inputs) 2025-11-03T16:36:17.5014279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5014342Z outputs = self.bert( 2025-11-03T16:36:17.5014625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5014705Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5014976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5015051Z layer_outputs = layer_module( 2025-11-03T16:36:17.5015266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5015348Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5015574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5015642Z return func(*args, **kwargs) 2025-11-03T16:36:17.5015929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5016006Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5016264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5016329Z return func(*args, **kwargs) 2025-11-03T16:36:17.5016592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5016664Z self_outputs = self.self( 2025-11-03T16:36:17.5016908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5016981Z return func(*args, **kwargs) 2025-11-03T16:36:17.5017251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5017332Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5017336Z 2025-11-03T16:36:17.5017411Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5017486Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5017589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5017773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5017840Z res = mod(**inputs) 2025-11-03T16:36:17.5018112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5018174Z outputs = self.bert( 2025-11-03T16:36:17.5018449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5018545Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5018818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5018884Z layer_outputs = layer_module( 2025-11-03T16:36:17.5019093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5019175Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5019396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5019469Z return func(*args, **kwargs) 2025-11-03T16:36:17.5019745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5019849Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5020075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5020137Z return func(*args, **kwargs) 2025-11-03T16:36:17.5020410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5020530Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5020807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5020886Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5020890Z 2025-11-03T16:36:17.5020987Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5021177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5021239Z res = mod(**inputs) 2025-11-03T16:36:17.5021516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5021578Z outputs = self.bert( 2025-11-03T16:36:17.5021865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5021934Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5022195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5022269Z layer_outputs = layer_module( 2025-11-03T16:36:17.5022489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5022571Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5022793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5022859Z return func(*args, **kwargs) 2025-11-03T16:36:17.5023133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5023211Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5023460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5023529Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5023819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5023926Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5024191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5024294Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5024297Z 2025-11-03T16:36:17.5024393Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5024583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5024644Z res = mod(**inputs) 2025-11-03T16:36:17.5024914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5024982Z outputs = self.bert( 2025-11-03T16:36:17.5025244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5025318Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5025586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5025674Z layer_outputs = layer_module( 2025-11-03T16:36:17.5025880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5025953Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5026188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5026250Z return func(*args, **kwargs) 2025-11-03T16:36:17.5026523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5026600Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5026840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5026918Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5027212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5027317Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5027603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5027716Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5027916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5027981Z return self.act(input) 2025-11-03T16:36:17.5027984Z 2025-11-03T16:36:17.5028089Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5028333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5028403Z res = mod(**inputs) 2025-11-03T16:36:17.5028674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5028736Z outputs = self.bert( 2025-11-03T16:36:17.5029011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5029078Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5029349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5029414Z layer_outputs = layer_module( 2025-11-03T16:36:17.5029615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5029698Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5029924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5030013Z return func(*args, **kwargs) 2025-11-03T16:36:17.5030282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5030364Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5030610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5030678Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5030976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5031099Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5031372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5031461Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5031466Z 2025-11-03T16:36:17.5031566Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5031750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5031811Z res = mod(**inputs) 2025-11-03T16:36:17.5032098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5032163Z outputs = self.bert( 2025-11-03T16:36:17.5032444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5032511Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5032826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5032903Z layer_outputs = layer_module( 2025-11-03T16:36:17.5033116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5033195Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5033432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5033510Z return func(*args, **kwargs) 2025-11-03T16:36:17.5033798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5033875Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5034187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5034260Z return func(*args, **kwargs) 2025-11-03T16:36:17.5034544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5034617Z self_outputs = self.self( 2025-11-03T16:36:17.5034851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5034924Z return func(*args, **kwargs) 2025-11-03T16:36:17.5035269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5035357Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5035361Z 2025-11-03T16:36:17.5035458Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5035650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5035721Z res = mod(**inputs) 2025-11-03T16:36:17.5035998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5036092Z outputs = self.bert( 2025-11-03T16:36:17.5036370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5036459Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5036736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5036804Z layer_outputs = layer_module( 2025-11-03T16:36:17.5037027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5037101Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5037340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5037405Z return func(*args, **kwargs) 2025-11-03T16:36:17.5037706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5037796Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5038027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5038100Z return func(*args, **kwargs) 2025-11-03T16:36:17.5038375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5038440Z self_outputs = self.self( 2025-11-03T16:36:17.5038678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5038744Z return func(*args, **kwargs) 2025-11-03T16:36:17.5039028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5039107Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5039111Z 2025-11-03T16:36:17.5039216Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5039401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5039461Z res = mod(**inputs) 2025-11-03T16:36:17.5039771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5039835Z outputs = self.bert( 2025-11-03T16:36:17.5040119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5040186Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5040476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5040554Z layer_outputs = layer_module( 2025-11-03T16:36:17.5040766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5040846Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5041080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5041150Z return func(*args, **kwargs) 2025-11-03T16:36:17.5041426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5041502Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5041742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5041807Z return func(*args, **kwargs) 2025-11-03T16:36:17.5042089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5042173Z self_outputs = self.self( 2025-11-03T16:36:17.5042410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5042481Z return func(*args, **kwargs) 2025-11-03T16:36:17.5042762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5042846Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5042849Z 2025-11-03T16:36:17.5042925Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5043008Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5043107Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5043295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5043379Z res = mod(**inputs) 2025-11-03T16:36:17.5043667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5043736Z outputs = self.bert( 2025-11-03T16:36:17.5044018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5044085Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5044362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5044428Z layer_outputs = layer_module( 2025-11-03T16:36:17.5044648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5044720Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5044959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5045035Z return func(*args, **kwargs) 2025-11-03T16:36:17.5045359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5045441Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5045677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5045746Z return func(*args, **kwargs) 2025-11-03T16:36:17.5046012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5046147Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5046420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5046497Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5046501Z 2025-11-03T16:36:17.5046602Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5046786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5046846Z res = mod(**inputs) 2025-11-03T16:36:17.5047123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5047183Z outputs = self.bert( 2025-11-03T16:36:17.5047453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5047522Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5047793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5047878Z layer_outputs = layer_module( 2025-11-03T16:36:17.5048084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5048164Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5048388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5048460Z return func(*args, **kwargs) 2025-11-03T16:36:17.5048727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5048805Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5049055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5049125Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5049428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5049551Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5049823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5049901Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5049904Z 2025-11-03T16:36:17.5049999Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5050189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5050250Z res = mod(**inputs) 2025-11-03T16:36:17.5050527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5050589Z outputs = self.bert( 2025-11-03T16:36:17.5050855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5050933Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5051198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5051285Z layer_outputs = layer_module( 2025-11-03T16:36:17.5051498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5051570Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5051805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5051882Z return func(*args, **kwargs) 2025-11-03T16:36:17.5052157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5052237Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5052488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5052558Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5052850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5052954Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5053217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5053329Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5053529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5053596Z return self.act(input) 2025-11-03T16:36:17.5053623Z 2025-11-03T16:36:17.5053720Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5053904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5053972Z res = mod(**inputs) 2025-11-03T16:36:17.5054245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5054312Z outputs = self.bert( 2025-11-03T16:36:17.5054576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5054642Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5054919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5054984Z layer_outputs = layer_module( 2025-11-03T16:36:17.5055214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5055287Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5055515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5055590Z return func(*args, **kwargs) 2025-11-03T16:36:17.5055860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5055943Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5056190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5056268Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5056569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5056695Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5056975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5057077Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5057081Z 2025-11-03T16:36:17.5057185Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5057366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5057428Z res = mod(**inputs) 2025-11-03T16:36:17.5057719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5057782Z outputs = self.bert( 2025-11-03T16:36:17.5058058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5058130Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5058406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5058473Z layer_outputs = layer_module( 2025-11-03T16:36:17.5058683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5058765Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5058990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5059061Z return func(*args, **kwargs) 2025-11-03T16:36:17.5059339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5059418Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5059689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5059758Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5060064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5060189Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5060467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.5060539Z return input_tensor + hidden_states 2025-11-03T16:36:17.5060543Z 2025-11-03T16:36:17.5060640Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5060834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5060913Z res = mod(**inputs) 2025-11-03T16:36:17.5061201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5061262Z outputs = self.bert( 2025-11-03T16:36:17.5061541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5061618Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5061890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5061966Z layer_outputs = layer_module( 2025-11-03T16:36:17.5062180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5062262Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5062493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5062560Z return func(*args, **kwargs) 2025-11-03T16:36:17.5062850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5062943Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5063179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5063245Z return func(*args, **kwargs) 2025-11-03T16:36:17.5063518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5063605Z self_outputs = self.self( 2025-11-03T16:36:17.5063837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5063912Z return func(*args, **kwargs) 2025-11-03T16:36:17.5064187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5064271Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5064275Z 2025-11-03T16:36:17.5064373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5064558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5064628Z res = mod(**inputs) 2025-11-03T16:36:17.5064902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5064972Z outputs = self.bert( 2025-11-03T16:36:17.5065245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5065317Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5065622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5065691Z layer_outputs = layer_module( 2025-11-03T16:36:17.5065912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5065986Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5066217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5066291Z return func(*args, **kwargs) 2025-11-03T16:36:17.5066565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5066652Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5066880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5066979Z return func(*args, **kwargs) 2025-11-03T16:36:17.5067256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5067322Z self_outputs = self.self( 2025-11-03T16:36:17.5067564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5067628Z return func(*args, **kwargs) 2025-11-03T16:36:17.5067911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5067984Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5067989Z 2025-11-03T16:36:17.5068087Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5068283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5068346Z res = mod(**inputs) 2025-11-03T16:36:17.5068633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5068695Z outputs = self.bert( 2025-11-03T16:36:17.5068989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5069059Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5069334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5069409Z layer_outputs = layer_module( 2025-11-03T16:36:17.5069636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5069721Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5069954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5070018Z return func(*args, **kwargs) 2025-11-03T16:36:17.5070301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5070379Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5070615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5070680Z return func(*args, **kwargs) 2025-11-03T16:36:17.5070968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5071036Z self_outputs = self.self( 2025-11-03T16:36:17.5071265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5071360Z return func(*args, **kwargs) 2025-11-03T16:36:17.5071636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5071717Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5071721Z 2025-11-03T16:36:17.5071800Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5071877Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5071979Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5072166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5072234Z res = mod(**inputs) 2025-11-03T16:36:17.5072513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5072577Z outputs = self.bert( 2025-11-03T16:36:17.5072874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5072944Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5073225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5073292Z layer_outputs = layer_module( 2025-11-03T16:36:17.5073508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5073581Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5073817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5073892Z return func(*args, **kwargs) 2025-11-03T16:36:17.5074229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5074326Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5074565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5074630Z return func(*args, **kwargs) 2025-11-03T16:36:17.5074954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5075084Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5075373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5075455Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5075476Z 2025-11-03T16:36:17.5075586Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5075779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5075845Z res = mod(**inputs) 2025-11-03T16:36:17.5076154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5076218Z outputs = self.bert( 2025-11-03T16:36:17.5076505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5076578Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5076853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5076931Z layer_outputs = layer_module( 2025-11-03T16:36:17.5077143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5077225Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5077475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5077539Z return func(*args, **kwargs) 2025-11-03T16:36:17.5077825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5077908Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5078168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5078241Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5078549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5078648Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5078921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5079029Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5079032Z 2025-11-03T16:36:17.5079131Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5079329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5079392Z res = mod(**inputs) 2025-11-03T16:36:17.5079670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5079741Z outputs = self.bert( 2025-11-03T16:36:17.5080023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5080102Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5080377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5080459Z layer_outputs = layer_module( 2025-11-03T16:36:17.5080673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5080749Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5081004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5081071Z return func(*args, **kwargs) 2025-11-03T16:36:17.5081353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5081432Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5081698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5081781Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5082082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5082189Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5082468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5082582Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5082786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5082857Z return self.act(input) 2025-11-03T16:36:17.5082860Z 2025-11-03T16:36:17.5082973Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5083161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5083232Z res = mod(**inputs) 2025-11-03T16:36:17.5083529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5083591Z outputs = self.bert( 2025-11-03T16:36:17.5083873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5083942Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5084230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5084298Z layer_outputs = layer_module( 2025-11-03T16:36:17.5084521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5084596Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5084828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5084934Z return func(*args, **kwargs) 2025-11-03T16:36:17.5085210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5085298Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5085547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5085620Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5085935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5086063Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5086345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5086426Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5086429Z 2025-11-03T16:36:17.5086535Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5086728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5086804Z res = mod(**inputs) 2025-11-03T16:36:17.5087093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5087155Z outputs = self.bert( 2025-11-03T16:36:17.5087433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5087516Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5087791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5087868Z layer_outputs = layer_module( 2025-11-03T16:36:17.5088082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5088177Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5088403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5088475Z return func(*args, **kwargs) 2025-11-03T16:36:17.5088743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5088818Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5089048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5089112Z return func(*args, **kwargs) 2025-11-03T16:36:17.5089386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5089467Z self_outputs = self.self( 2025-11-03T16:36:17.5089692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5089762Z return func(*args, **kwargs) 2025-11-03T16:36:17.5090032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5090115Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5090118Z 2025-11-03T16:36:17.5090213Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5090406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5090465Z res = mod(**inputs) 2025-11-03T16:36:17.5090739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5090828Z outputs = self.bert( 2025-11-03T16:36:17.5091095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5091169Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5091437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5091503Z layer_outputs = layer_module( 2025-11-03T16:36:17.5091719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5091793Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5092025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5092091Z return func(*args, **kwargs) 2025-11-03T16:36:17.5092370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5092454Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5092701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5092774Z return func(*args, **kwargs) 2025-11-03T16:36:17.5093041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5093112Z self_outputs = self.self( 2025-11-03T16:36:17.5093358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5093422Z return func(*args, **kwargs) 2025-11-03T16:36:17.5093696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5093773Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5093776Z 2025-11-03T16:36:17.5093876Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5094061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5094123Z res = mod(**inputs) 2025-11-03T16:36:17.5094400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5094461Z outputs = self.bert( 2025-11-03T16:36:17.5094733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5094801Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5095074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5095161Z layer_outputs = layer_module( 2025-11-03T16:36:17.5095374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5095455Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5095681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5095752Z return func(*args, **kwargs) 2025-11-03T16:36:17.5096016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5096092Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5096323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5096385Z return func(*args, **kwargs) 2025-11-03T16:36:17.5096680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5096746Z self_outputs = self.self( 2025-11-03T16:36:17.5096975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5097045Z return func(*args, **kwargs) 2025-11-03T16:36:17.5097317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5097398Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5097401Z 2025-11-03T16:36:17.5097475Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5097555Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5097651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5097833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5097904Z res = mod(**inputs) 2025-11-03T16:36:17.5098175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5098241Z outputs = self.bert( 2025-11-03T16:36:17.5098522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5098590Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5098866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5098935Z layer_outputs = layer_module( 2025-11-03T16:36:17.5099163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5099238Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5099469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5099531Z return func(*args, **kwargs) 2025-11-03T16:36:17.5099797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5099880Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5100102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5100172Z return func(*args, **kwargs) 2025-11-03T16:36:17.5100440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5100559Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5100831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5100926Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5100931Z 2025-11-03T16:36:17.5101030Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5101210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5101277Z res = mod(**inputs) 2025-11-03T16:36:17.5101543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5101604Z outputs = self.bert( 2025-11-03T16:36:17.5101876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5101944Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5102216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5102299Z layer_outputs = layer_module( 2025-11-03T16:36:17.5102502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5102583Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5102807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5102875Z return func(*args, **kwargs) 2025-11-03T16:36:17.5103140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5103217Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5103465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5103536Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5103849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5103945Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5104231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5104308Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5104311Z 2025-11-03T16:36:17.5104405Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5104595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5104655Z res = mod(**inputs) 2025-11-03T16:36:17.5104945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5105009Z outputs = self.bert( 2025-11-03T16:36:17.5105276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5105350Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5105617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5105690Z layer_outputs = layer_module( 2025-11-03T16:36:17.5105896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5105974Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5106198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5106263Z return func(*args, **kwargs) 2025-11-03T16:36:17.5106535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5106632Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5106880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5106951Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5107245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5107349Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5107616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5107730Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5107930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5108021Z return self.act(input) 2025-11-03T16:36:17.5108024Z 2025-11-03T16:36:17.5108120Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5108305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5108375Z res = mod(**inputs) 2025-11-03T16:36:17.5108649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5108719Z outputs = self.bert( 2025-11-03T16:36:17.5108989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5109057Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5109334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5109404Z layer_outputs = layer_module( 2025-11-03T16:36:17.5109616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5109689Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5109938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5110004Z return func(*args, **kwargs) 2025-11-03T16:36:17.5110271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5110355Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5110613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5110690Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5110990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5111115Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5111390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5111465Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5111468Z 2025-11-03T16:36:17.5111569Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5111753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5111819Z res = mod(**inputs) 2025-11-03T16:36:17.5112090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5112152Z outputs = self.bert( 2025-11-03T16:36:17.5112424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5112511Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5112792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5112860Z layer_outputs = layer_module( 2025-11-03T16:36:17.5113076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5113160Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5113564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5113646Z return func(*args, **kwargs) 2025-11-03T16:36:17.5113923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5114131Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5114391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5114465Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5114782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5114913Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5115202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.5115288Z return input_tensor + hidden_states 2025-11-03T16:36:17.5115293Z 2025-11-03T16:36:17.5115392Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5115588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5115653Z res = mod(**inputs) 2025-11-03T16:36:17.5115939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5116002Z outputs = self.bert( 2025-11-03T16:36:17.5116327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5116401Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5116673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5116750Z layer_outputs = layer_module( 2025-11-03T16:36:17.5116985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5117069Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5117300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5117364Z return func(*args, **kwargs) 2025-11-03T16:36:17.5117645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5117722Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5117968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5118032Z return func(*args, **kwargs) 2025-11-03T16:36:17.5118313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5118379Z self_outputs = self.self( 2025-11-03T16:36:17.5118609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5118710Z return func(*args, **kwargs) 2025-11-03T16:36:17.5118990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5119079Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5119082Z 2025-11-03T16:36:17.5119183Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5119372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5119440Z res = mod(**inputs) 2025-11-03T16:36:17.5119718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5119789Z outputs = self.bert( 2025-11-03T16:36:17.5120064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5120157Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5120447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5120512Z layer_outputs = layer_module( 2025-11-03T16:36:17.5120733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5120806Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5121043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5121108Z return func(*args, **kwargs) 2025-11-03T16:36:17.5121386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5121470Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5121703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5121776Z return func(*args, **kwargs) 2025-11-03T16:36:17.5122050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5122171Z self_outputs = self.self( 2025-11-03T16:36:17.5122413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5122478Z return func(*args, **kwargs) 2025-11-03T16:36:17.5122763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5122854Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5122858Z 2025-11-03T16:36:17.5122963Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5123157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5123219Z res = mod(**inputs) 2025-11-03T16:36:17.5123509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5123572Z outputs = self.bert( 2025-11-03T16:36:17.5123864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5123932Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5124205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5124282Z layer_outputs = layer_module( 2025-11-03T16:36:17.5124497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5124581Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5124825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5124889Z return func(*args, **kwargs) 2025-11-03T16:36:17.5125170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5125251Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5125487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5125551Z return func(*args, **kwargs) 2025-11-03T16:36:17.5125832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5125897Z self_outputs = self.self( 2025-11-03T16:36:17.5126131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5126221Z return func(*args, **kwargs) 2025-11-03T16:36:17.5126496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5126580Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5126585Z 2025-11-03T16:36:17.5126663Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5126738Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5126844Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5127031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5127100Z res = mod(**inputs) 2025-11-03T16:36:17.5127379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5127450Z outputs = self.bert( 2025-11-03T16:36:17.5127725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5127793Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5128089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5128157Z layer_outputs = layer_module( 2025-11-03T16:36:17.5128428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5128501Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5128738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5128812Z return func(*args, **kwargs) 2025-11-03T16:36:17.5129078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5129163Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5129385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5129448Z return func(*args, **kwargs) 2025-11-03T16:36:17.5129724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5129844Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5130117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5130196Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5130199Z 2025-11-03T16:36:17.5130301Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5130485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5130560Z res = mod(**inputs) 2025-11-03T16:36:17.5130835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5130896Z outputs = self.bert( 2025-11-03T16:36:17.5131168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5131235Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5131499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5131571Z layer_outputs = layer_module( 2025-11-03T16:36:17.5131776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5131872Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5132096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5132165Z return func(*args, **kwargs) 2025-11-03T16:36:17.5132433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5132510Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5132762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5132832Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5133132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5133228Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5133495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5133576Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5133579Z 2025-11-03T16:36:17.5133674Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5133884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5133946Z res = mod(**inputs) 2025-11-03T16:36:17.5134220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5134281Z outputs = self.bert( 2025-11-03T16:36:17.5134558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5134635Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5134908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5134982Z layer_outputs = layer_module( 2025-11-03T16:36:17.5135191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5135265Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5135505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5135569Z return func(*args, **kwargs) 2025-11-03T16:36:17.5135843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5135922Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5136175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5136266Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5136563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5136667Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5136936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5137048Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5137250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5137315Z return self.act(input) 2025-11-03T16:36:17.5137326Z 2025-11-03T16:36:17.5137421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5137603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5137699Z res = mod(**inputs) 2025-11-03T16:36:17.5137971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5138041Z outputs = self.bert( 2025-11-03T16:36:17.5138308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5138375Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5138649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5138714Z layer_outputs = layer_module( 2025-11-03T16:36:17.5138927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5139000Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5139226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5139300Z return func(*args, **kwargs) 2025-11-03T16:36:17.5139565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5139668Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5139911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5139988Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5140294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5140418Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5140694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5140772Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5140775Z 2025-11-03T16:36:17.5140876Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5141062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5141123Z res = mod(**inputs) 2025-11-03T16:36:17.5141399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5141460Z outputs = self.bert( 2025-11-03T16:36:17.5141733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5141799Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5142070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5142182Z layer_outputs = layer_module( 2025-11-03T16:36:17.5142388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5142467Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5142692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5142763Z return func(*args, **kwargs) 2025-11-03T16:36:17.5143031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5143106Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5143338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5143419Z return func(*args, **kwargs) 2025-11-03T16:36:17.5143696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5143761Z self_outputs = self.self( 2025-11-03T16:36:17.5143987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5144057Z return func(*args, **kwargs) 2025-11-03T16:36:17.5144325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5144408Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5144411Z 2025-11-03T16:36:17.5144505Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5144698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5144756Z res = mod(**inputs) 2025-11-03T16:36:17.5145035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5145107Z outputs = self.bert( 2025-11-03T16:36:17.5145380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5145474Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5145742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5145807Z layer_outputs = layer_module( 2025-11-03T16:36:17.5146022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5146112Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5146347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5146413Z return func(*args, **kwargs) 2025-11-03T16:36:17.5146686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5146762Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5146985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5147058Z return func(*args, **kwargs) 2025-11-03T16:36:17.5147326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5147396Z self_outputs = self.self( 2025-11-03T16:36:17.5147623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5147684Z return func(*args, **kwargs) 2025-11-03T16:36:17.5147960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5148053Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5148056Z 2025-11-03T16:36:17.5148157Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5148342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5148408Z res = mod(**inputs) 2025-11-03T16:36:17.5148679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5148740Z outputs = self.bert( 2025-11-03T16:36:17.5149018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5149086Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5149362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5149442Z layer_outputs = layer_module( 2025-11-03T16:36:17.5149647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5149729Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5149952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5150021Z return func(*args, **kwargs) 2025-11-03T16:36:17.5150289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5150365Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5150595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5150659Z return func(*args, **kwargs) 2025-11-03T16:36:17.5150931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5150995Z self_outputs = self.self( 2025-11-03T16:36:17.5151241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5151306Z return func(*args, **kwargs) 2025-11-03T16:36:17.5151579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5151665Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5151668Z 2025-11-03T16:36:17.5151745Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5151846Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5151947Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5152135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5152208Z res = mod(**inputs) 2025-11-03T16:36:17.5152485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5152555Z outputs = self.bert( 2025-11-03T16:36:17.5152831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5152900Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5153179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5153248Z layer_outputs = layer_module( 2025-11-03T16:36:17.5153473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5153549Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5153804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5153869Z return func(*args, **kwargs) 2025-11-03T16:36:17.5154212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5154305Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5154541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5154616Z return func(*args, **kwargs) 2025-11-03T16:36:17.5154898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5155025Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5155341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5155423Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5155427Z 2025-11-03T16:36:17.5155534Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5155728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5155799Z res = mod(**inputs) 2025-11-03T16:36:17.5156082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5156145Z outputs = self.bert( 2025-11-03T16:36:17.5156434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5156504Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5156794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5156866Z layer_outputs = layer_module( 2025-11-03T16:36:17.5157081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5157184Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5157417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5157492Z return func(*args, **kwargs) 2025-11-03T16:36:17.5157766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5157870Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5158117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5158191Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5158510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5158608Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5158888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5158967Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5158971Z 2025-11-03T16:36:17.5159072Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5159259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5159323Z res = mod(**inputs) 2025-11-03T16:36:17.5159606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5159696Z outputs = self.bert( 2025-11-03T16:36:17.5159982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5160053Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5160334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5160409Z layer_outputs = layer_module( 2025-11-03T16:36:17.5160622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5160701Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5160936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5160999Z return func(*args, **kwargs) 2025-11-03T16:36:17.5161300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5161379Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5161643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5161716Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5162023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5162121Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5162399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5162514Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5162719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5162797Z return self.act(input) 2025-11-03T16:36:17.5162801Z 2025-11-03T16:36:17.5162898Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5163087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5163170Z res = mod(**inputs) 2025-11-03T16:36:17.5163448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5163518Z outputs = self.bert( 2025-11-03T16:36:17.5163792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5163885Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5164161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5164230Z layer_outputs = layer_module( 2025-11-03T16:36:17.5164448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5164523Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5164765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5164834Z return func(*args, **kwargs) 2025-11-03T16:36:17.5165110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5165197Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5165450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5165529Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5165845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5165977Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5166253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5166331Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5166335Z 2025-11-03T16:36:17.5166439Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5166625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5166692Z res = mod(**inputs) 2025-11-03T16:36:17.5166974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5167053Z outputs = self.bert( 2025-11-03T16:36:17.5167338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5167410Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5167696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5167764Z layer_outputs = layer_module( 2025-11-03T16:36:17.5167984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5168061Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5168292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5168366Z return func(*args, **kwargs) 2025-11-03T16:36:17.5168640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5168730Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5168977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5169047Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5169371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5169497Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5169780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.5169869Z return input_tensor + hidden_states 2025-11-03T16:36:17.5169873Z 2025-11-03T16:36:17.5169979Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5170171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5170236Z res = mod(**inputs) 2025-11-03T16:36:17.5170524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5170590Z outputs = self.bert( 2025-11-03T16:36:17.5170876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5170946Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5171221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5171300Z layer_outputs = layer_module( 2025-11-03T16:36:17.5171513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5171615Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5171846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5171918Z return func(*args, **kwargs) 2025-11-03T16:36:17.5172194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5172270Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5172507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5172572Z return func(*args, **kwargs) 2025-11-03T16:36:17.5172854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5172922Z self_outputs = self.self( 2025-11-03T16:36:17.5173168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5173242Z return func(*args, **kwargs) 2025-11-03T16:36:17.5173518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5173606Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5173609Z 2025-11-03T16:36:17.5173707Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5173896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5173964Z res = mod(**inputs) 2025-11-03T16:36:17.5174245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5174317Z outputs = self.bert( 2025-11-03T16:36:17.5174592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5174671Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5174946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5175011Z layer_outputs = layer_module( 2025-11-03T16:36:17.5175246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5175321Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5175557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5175621Z return func(*args, **kwargs) 2025-11-03T16:36:17.5175912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5175997Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5176226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5176296Z return func(*args, **kwargs) 2025-11-03T16:36:17.5176575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5176649Z self_outputs = self.self( 2025-11-03T16:36:17.5176876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5176941Z return func(*args, **kwargs) 2025-11-03T16:36:17.5177223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5177297Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5177301Z 2025-11-03T16:36:17.5177405Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5177609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5177669Z res = mod(**inputs) 2025-11-03T16:36:17.5177955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5178017Z outputs = self.bert( 2025-11-03T16:36:17.5178297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5178365Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5178643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5178711Z layer_outputs = layer_module( 2025-11-03T16:36:17.5178921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5179021Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5179249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5179318Z return func(*args, **kwargs) 2025-11-03T16:36:17.5179597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5179675Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5179910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5179972Z return func(*args, **kwargs) 2025-11-03T16:36:17.5180257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5180323Z self_outputs = self.self( 2025-11-03T16:36:17.5180554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5180623Z return func(*args, **kwargs) 2025-11-03T16:36:17.5180899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5180998Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5181002Z 2025-11-03T16:36:17.5181080Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5181162Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5181258Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5181444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5181538Z res = mod(**inputs) 2025-11-03T16:36:17.5181821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5181895Z outputs = self.bert( 2025-11-03T16:36:17.5182168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5182238Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5182524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5182592Z layer_outputs = layer_module( 2025-11-03T16:36:17.5182821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5182910Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5183147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5183221Z return func(*args, **kwargs) 2025-11-03T16:36:17.5183498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5183602Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5183832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5183905Z return func(*args, **kwargs) 2025-11-03T16:36:17.5184178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5184300Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5184584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5184665Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5184668Z 2025-11-03T16:36:17.5184798Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5184986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5185046Z res = mod(**inputs) 2025-11-03T16:36:17.5185331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5185395Z outputs = self.bert( 2025-11-03T16:36:17.5185677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5185744Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5186029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5186095Z layer_outputs = layer_module( 2025-11-03T16:36:17.5186304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5186390Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5186617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5186689Z return func(*args, **kwargs) 2025-11-03T16:36:17.5186983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5187065Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5187319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5187391Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5187712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5187815Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5188106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5188184Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5188187Z 2025-11-03T16:36:17.5188282Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5188476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5188536Z res = mod(**inputs) 2025-11-03T16:36:17.5188814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5188877Z outputs = self.bert( 2025-11-03T16:36:17.5189145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5189221Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5189506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5189577Z layer_outputs = layer_module( 2025-11-03T16:36:17.5189788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5189865Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5190090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5190154Z return func(*args, **kwargs) 2025-11-03T16:36:17.5190430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5190506Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5190753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5190840Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5191134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5191237Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5191503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5191615Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5191814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5191888Z return self.act(input) 2025-11-03T16:36:17.5191891Z 2025-11-03T16:36:17.5191984Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5192169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5192240Z res = mod(**inputs) 2025-11-03T16:36:17.5192507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5192574Z outputs = self.bert( 2025-11-03T16:36:17.5192856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5192924Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5193196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5193262Z layer_outputs = layer_module( 2025-11-03T16:36:17.5193486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5193562Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5193800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5193865Z return func(*args, **kwargs) 2025-11-03T16:36:17.5194206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5194298Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5194551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5194633Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5194939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5195075Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5195374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5195452Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5195456Z 2025-11-03T16:36:17.5195561Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5195748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5195819Z res = mod(**inputs) 2025-11-03T16:36:17.5196089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5196152Z outputs = self.bert( 2025-11-03T16:36:17.5196427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5196497Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5196785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5196852Z layer_outputs = layer_module( 2025-11-03T16:36:17.5197062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5197148Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5197371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5197446Z return func(*args, **kwargs) 2025-11-03T16:36:17.5197712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5197790Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5198020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5198087Z return func(*args, **kwargs) 2025-11-03T16:36:17.5198360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5198425Z self_outputs = self.self( 2025-11-03T16:36:17.5198675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5198739Z return func(*args, **kwargs) 2025-11-03T16:36:17.5199007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 222, in forward 2025-11-03T16:36:17.5199091Z query_layer = self.query(hidden_states) 2025-11-03T16:36:17.5199095Z 2025-11-03T16:36:17.5199203Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5199395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5199460Z res = mod(**inputs) 2025-11-03T16:36:17.5199733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5199801Z outputs = self.bert( 2025-11-03T16:36:17.5200070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5200145Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5200414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5200487Z layer_outputs = layer_module( 2025-11-03T16:36:17.5200698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5200770Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5201005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5201088Z return func(*args, **kwargs) 2025-11-03T16:36:17.5201360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5201436Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5201660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5201730Z return func(*args, **kwargs) 2025-11-03T16:36:17.5202001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5202074Z self_outputs = self.self( 2025-11-03T16:36:17.5202297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5202386Z return func(*args, **kwargs) 2025-11-03T16:36:17.5202651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-11-03T16:36:17.5202723Z key_layer = self.key(current_states) 2025-11-03T16:36:17.5202727Z 2025-11-03T16:36:17.5202830Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5203014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5203080Z res = mod(**inputs) 2025-11-03T16:36:17.5203349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5203408Z outputs = self.bert( 2025-11-03T16:36:17.5203682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5203748Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5204019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5204086Z layer_outputs = layer_module( 2025-11-03T16:36:17.5204290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5204396Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5204618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5204689Z return func(*args, **kwargs) 2025-11-03T16:36:17.5204969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5205052Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5205277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5205342Z return func(*args, **kwargs) 2025-11-03T16:36:17.5205613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-11-03T16:36:17.5205677Z self_outputs = self.self( 2025-11-03T16:36:17.5205905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5205967Z return func(*args, **kwargs) 2025-11-03T16:36:17.5206231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-11-03T16:36:17.5206311Z value_layer = self.value(current_states) 2025-11-03T16:36:17.5206314Z 2025-11-03T16:36:17.5206389Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5206470Z cudagraph partition due to non gpu ops 2025-11-03T16:36:17.5206567Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5206774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5206834Z res = mod(**inputs) 2025-11-03T16:36:17.5207104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5207173Z outputs = self.bert( 2025-11-03T16:36:17.5207438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5207512Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5207777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5207842Z layer_outputs = layer_module( 2025-11-03T16:36:17.5208056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5208147Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5208375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5208436Z return func(*args, **kwargs) 2025-11-03T16:36:17.5208701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-11-03T16:36:17.5208783Z self_attention_outputs = self.attention( 2025-11-03T16:36:17.5209005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5209076Z return func(*args, **kwargs) 2025-11-03T16:36:17.5209343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-11-03T16:36:17.5209467Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:36:17.5209735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-11-03T16:36:17.5209813Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5209816Z 2025-11-03T16:36:17.5209919Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5210116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5210186Z res = mod(**inputs) 2025-11-03T16:36:17.5210460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5210520Z outputs = self.bert( 2025-11-03T16:36:17.5210805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5210871Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5211146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5211211Z layer_outputs = layer_module( 2025-11-03T16:36:17.5211428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5211502Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5211727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5211798Z return func(*args, **kwargs) 2025-11-03T16:36:17.5212073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5212166Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5212407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5212493Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5212795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5212895Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5213169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-11-03T16:36:17.5213394Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5213400Z 2025-11-03T16:36:17.5213509Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5213695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5213757Z res = mod(**inputs) 2025-11-03T16:36:17.5214037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5214179Z outputs = self.bert( 2025-11-03T16:36:17.5214454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5214522Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5214788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5214861Z layer_outputs = layer_module( 2025-11-03T16:36:17.5215068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5215149Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5215378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5215453Z return func(*args, **kwargs) 2025-11-03T16:36:17.5215720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5215799Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5216072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5216145Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5216445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-11-03T16:36:17.5216541Z intermediate_output = self.intermediate(ln_output) 2025-11-03T16:36:17.5216827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-11-03T16:36:17.5216941Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:36:17.5217146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:36:17.5217218Z return self.act(input) 2025-11-03T16:36:17.5217221Z 2025-11-03T16:36:17.5217317Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5217510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5217570Z res = mod(**inputs) 2025-11-03T16:36:17.5217843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5217913Z outputs = self.bert( 2025-11-03T16:36:17.5218178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5218252Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5218518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5218608Z layer_outputs = layer_module( 2025-11-03T16:36:17.5218826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5218900Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5219135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5219199Z return func(*args, **kwargs) 2025-11-03T16:36:17.5219465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5219551Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5219793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5219886Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5220183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5220313Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5220589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-11-03T16:36:17.5220664Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5220667Z 2025-11-03T16:36:17.5220769Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5220953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5221025Z res = mod(**inputs) 2025-11-03T16:36:17.5221302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-11-03T16:36:17.5221374Z outputs = self.bert( 2025-11-03T16:36:17.5221639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-11-03T16:36:17.5221706Z encoder_outputs = self.encoder( 2025-11-03T16:36:17.5221995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-11-03T16:36:17.5222062Z layer_outputs = layer_module( 2025-11-03T16:36:17.5222275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:36:17.5222348Z return super().__call__(*args, **kwargs) 2025-11-03T16:36:17.5222586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:36:17.5222661Z return func(*args, **kwargs) 2025-11-03T16:36:17.5222941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-11-03T16:36:17.5223025Z layer_output = apply_chunking_to_forward( 2025-11-03T16:36:17.5223267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:36:17.5223338Z return forward_fn(*input_tensors) 2025-11-03T16:36:17.5223639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-11-03T16:36:17.5223762Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:36:17.5224035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-11-03T16:36:17.5224107Z return input_tensor + hidden_states 2025-11-03T16:36:17.5224110Z 2025-11-03T16:36:17.5224215Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5224414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5224474Z res = mod(**inputs) 2025-11-03T16:36:17.5224753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-11-03T16:36:17.5224843Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:36:17.5225112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-11-03T16:36:17.5225216Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:36:17.5225492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 632, in forward 2025-11-03T16:36:17.5225580Z hidden_states = self.transform(hidden_states) 2025-11-03T16:36:17.5225870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 607, in forward 2025-11-03T16:36:17.5225956Z hidden_states = self.dense(hidden_states) 2025-11-03T16:36:17.5225959Z 2025-11-03T16:36:17.5226054Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5226244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5226303Z res = mod(**inputs) 2025-11-03T16:36:17.5226574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-11-03T16:36:17.5226669Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:36:17.5226936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-11-03T16:36:17.5227048Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:36:17.5227317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 633, in forward 2025-11-03T16:36:17.5227407Z hidden_states = self.decoder(hidden_states) 2025-11-03T16:36:17.5227410Z 2025-11-03T16:36:17.5227506Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:36:17.5227704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:36:17.5227775Z res = mod(**inputs) 2025-11-03T16:36:17.5228042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1078, in forward 2025-11-03T16:36:17.5228116Z lm_loss = self.loss_function( 2025-11-03T16:36:17.5228356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-11-03T16:36:17.5228523Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-11-03T16:36:17.5228779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-11-03T16:36:17.5228964Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-11-03T16:36:17.5228968Z 2025-11-03T16:36:28.3456272Z Compilation time (from dynamo_timed): 23.699360977 2025-11-03T16:36:28.3495858Z pass 2025-11-03T16:36:28.3498334Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:36:28.3499379Z TIMING: _recursive_pre_grad_passes:0.00968 _recursive_joint_graph_passes:1.06459 _recursive_post_grad_passes:0.09846 async_compile.wait:0.70734 code_gen:10.10756 inductor_compile:12.39553 backend_compile:18.13547 gc:0.00036 entire_frame_compile:23.69936 total_wall_time:23.69936 2025-11-03T16:36:28.3505281Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:15932 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:4866 2025-11-03T16:36:28.3507214Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-11-03T16:36:31.0323886Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:36:31.0324671Z import pynvml # type: ignore[import] 2025-11-03T16:36:34.2320280Z 2025-11-03T16:36:34.6859479Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:36:34.6861496Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:36:34.6916459Z cpu eval MobileBertForMaskedLM 2025-11-03T16:36:34.9563679Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:36:35.1114420Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:36:35.2641684Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:00.7816870Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.7821368Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.7826446Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7832381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7832965Z res = mod(**inputs) 2025-11-03T16:37:00.7833854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7834462Z outputs = self.mobilebert( 2025-11-03T16:37:00.7834932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-11-03T16:37:00.7835406Z embedding_output = self.embeddings( 2025-11-03T16:37:00.7835993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-11-03T16:37:00.7836403Z inputs_embeds = torch.cat( 2025-11-03T16:37:00.7836523Z 2025-11-03T16:37:00.7836645Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7837359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7837707Z res = mod(**inputs) 2025-11-03T16:37:00.7838099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-11-03T16:37:00.7838544Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:37:00.7839048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-11-03T16:37:00.7839538Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:37:00.7840037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-11-03T16:37:00.7840588Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-11-03T16:37:00.7840850Z 2025-11-03T16:37:00.7840970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7841333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7841667Z res = mod(**inputs) 2025-11-03T16:37:00.7842064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7842494Z outputs = self.mobilebert( 2025-11-03T16:37:00.7842901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-11-03T16:37:00.7843315Z embedding_output = self.embeddings( 2025-11-03T16:37:00.7843828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-11-03T16:37:00.7844299Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-11-03T16:37:00.7844478Z 2025-11-03T16:37:00.7844592Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7844959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7845278Z res = mod(**inputs) 2025-11-03T16:37:00.7845676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7846101Z outputs = self.mobilebert( 2025-11-03T16:37:00.7846517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-11-03T16:37:00.7846947Z embedding_output = self.embeddings( 2025-11-03T16:37:00.7847413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-11-03T16:37:00.7847835Z embeddings = self.LayerNorm(embeddings) 2025-11-03T16:37:00.7848264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.7848700Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.7848851Z 2025-11-03T16:37:00.7848956Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7849318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7849640Z res = mod(**inputs) 2025-11-03T16:37:00.7850035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7850450Z outputs = self.mobilebert( 2025-11-03T16:37:00.7850850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7851274Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7851696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7852172Z layer_outputs = layer_module( 2025-11-03T16:37:00.7852619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.7853169Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.7853758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.7854213Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.7854664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.7855094Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.7855234Z 2025-11-03T16:37:00.7855339Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7855701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7856026Z res = mod(**inputs) 2025-11-03T16:37:00.7856415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7856822Z outputs = self.mobilebert( 2025-11-03T16:37:00.7857226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7857642Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7858064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7858509Z layer_outputs = layer_module( 2025-11-03T16:37:00.7858913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.7859346Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.7859801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.7860237Z self_outputs = self.self( 2025-11-03T16:37:00.7860662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.7861077Z self.value(value_tensor) 2025-11-03T16:37:00.7861196Z 2025-11-03T16:37:00.7861312Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7861681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7862020Z res = mod(**inputs) 2025-11-03T16:37:00.7862414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7862827Z outputs = self.mobilebert( 2025-11-03T16:37:00.7863229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7863636Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7864051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7864472Z layer_outputs = layer_module( 2025-11-03T16:37:00.7864898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.7865433Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.7865919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.7866364Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.7866839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.7867263Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.7867395Z 2025-11-03T16:37:00.7867502Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7867840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7868172Z res = mod(**inputs) 2025-11-03T16:37:00.7868559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7868986Z outputs = self.mobilebert( 2025-11-03T16:37:00.7869398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7869829Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7870255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7870682Z layer_outputs = layer_module( 2025-11-03T16:37:00.7871097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.7871607Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.7872125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.7872587Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.7873066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.7873510Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.7873980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.7874597Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.7874770Z 2025-11-03T16:37:00.7874881Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7875265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7875615Z res = mod(**inputs) 2025-11-03T16:37:00.7875995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7876432Z outputs = self.mobilebert( 2025-11-03T16:37:00.7876828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7877228Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7877635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7878039Z layer_outputs = layer_module( 2025-11-03T16:37:00.7878443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.7878858Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.7879272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.7879681Z self_outputs = self.self( 2025-11-03T16:37:00.7880078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.7880488Z self.query(query_tensor) 2025-11-03T16:37:00.7880607Z 2025-11-03T16:37:00.7880708Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7881080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7881397Z res = mod(**inputs) 2025-11-03T16:37:00.7881787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7882205Z outputs = self.mobilebert( 2025-11-03T16:37:00.7882619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7883088Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7883489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7883921Z layer_outputs = layer_module( 2025-11-03T16:37:00.7884349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.7884801Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.7885257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.7885686Z self_outputs = self.self( 2025-11-03T16:37:00.7886109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.7886542Z self.key(key_tensor) 2025-11-03T16:37:00.7886658Z 2025-11-03T16:37:00.7886755Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.7886987Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.7887241Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7887666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7888009Z res = mod(**inputs) 2025-11-03T16:37:00.7888423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7888854Z outputs = self.mobilebert( 2025-11-03T16:37:00.7889276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7889715Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7890147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7890582Z layer_outputs = layer_module( 2025-11-03T16:37:00.7890978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.7891426Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.7891835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.7892282Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.7892726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.7893143Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.7893284Z 2025-11-03T16:37:00.7893383Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7893733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7894048Z res = mod(**inputs) 2025-11-03T16:37:00.7894421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7894841Z outputs = self.mobilebert( 2025-11-03T16:37:00.7895233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7895634Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7896051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7896443Z layer_outputs = layer_module( 2025-11-03T16:37:00.7896840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.7897254Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.7897706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.7898171Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.7898626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.7899100Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.7899581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.7900015Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.7900168Z 2025-11-03T16:37:00.7900277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7900627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7900951Z res = mod(**inputs) 2025-11-03T16:37:00.7901337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7901773Z outputs = self.mobilebert( 2025-11-03T16:37:00.7902178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7902591Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7903003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7903430Z layer_outputs = layer_module( 2025-11-03T16:37:00.7903820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7904243Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7904685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.7905144Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.7905622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.7906047Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.7906189Z 2025-11-03T16:37:00.7906292Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7906653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7906975Z res = mod(**inputs) 2025-11-03T16:37:00.7907364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7907777Z outputs = self.mobilebert( 2025-11-03T16:37:00.7908170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7908584Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7908999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7909434Z layer_outputs = layer_module( 2025-11-03T16:37:00.7909849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7910329Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7910794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.7911276Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.7911772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.7912253Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.7912438Z 2025-11-03T16:37:00.7912552Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7912937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7913502Z res = mod(**inputs) 2025-11-03T16:37:00.7913931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7914432Z outputs = self.mobilebert( 2025-11-03T16:37:00.7914867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7915320Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7915741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7916153Z layer_outputs = layer_module( 2025-11-03T16:37:00.7916555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7917057Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7917498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.7917962Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.7918436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.7918864Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.7919009Z 2025-11-03T16:37:00.7919119Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7919484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7919803Z res = mod(**inputs) 2025-11-03T16:37:00.7920199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7920652Z outputs = self.mobilebert( 2025-11-03T16:37:00.7921058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7921464Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7921878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7922291Z layer_outputs = layer_module( 2025-11-03T16:37:00.7922706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7923140Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7923566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.7924037Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.7924505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.7925016Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.7925530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.7925965Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.7926115Z 2025-11-03T16:37:00.7926218Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7926579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7926933Z res = mod(**inputs) 2025-11-03T16:37:00.7927324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7927743Z outputs = self.mobilebert( 2025-11-03T16:37:00.7928144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7928562Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7928970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7929385Z layer_outputs = layer_module( 2025-11-03T16:37:00.7929793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7930219Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7930657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.7931095Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.7931556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.7931984Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.7932126Z 2025-11-03T16:37:00.7932227Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7932591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7932915Z res = mod(**inputs) 2025-11-03T16:37:00.7933304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7933713Z outputs = self.mobilebert( 2025-11-03T16:37:00.7934118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7934517Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7934933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7935337Z layer_outputs = layer_module( 2025-11-03T16:37:00.7935745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7936171Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7936597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.7937040Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.7937484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.7937919Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.7938088Z 2025-11-03T16:37:00.7938190Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7938545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7938863Z res = mod(**inputs) 2025-11-03T16:37:00.7939250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7939656Z outputs = self.mobilebert( 2025-11-03T16:37:00.7940046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7940445Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7940886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7941291Z layer_outputs = layer_module( 2025-11-03T16:37:00.7941696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7942138Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7942578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.7943100Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.7943560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.7943986Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.7944134Z 2025-11-03T16:37:00.7944238Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7944603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7944932Z res = mod(**inputs) 2025-11-03T16:37:00.7945352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7945791Z outputs = self.mobilebert( 2025-11-03T16:37:00.7946196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7946612Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7947018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7947432Z layer_outputs = layer_module( 2025-11-03T16:37:00.7947838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7948277Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7948738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.7949257Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.7949762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.7950254Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.7950756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.7951210Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.7951358Z 2025-11-03T16:37:00.7951461Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7951823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7952145Z res = mod(**inputs) 2025-11-03T16:37:00.7952533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7952980Z outputs = self.mobilebert( 2025-11-03T16:37:00.7953411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7953863Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7955125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7955577Z layer_outputs = layer_module( 2025-11-03T16:37:00.7956009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7956440Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7956920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.7957383Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.7957840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.7958269Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.7958413Z 2025-11-03T16:37:00.7958519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7958879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7959204Z res = mod(**inputs) 2025-11-03T16:37:00.7959596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7960004Z outputs = self.mobilebert( 2025-11-03T16:37:00.7960412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7960828Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7961264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7961683Z layer_outputs = layer_module( 2025-11-03T16:37:00.7962077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7962504Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7962935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.7963399Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.7963858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.7964313Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.7964508Z 2025-11-03T16:37:00.7964611Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7964967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7965289Z res = mod(**inputs) 2025-11-03T16:37:00.7965690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7966115Z outputs = self.mobilebert( 2025-11-03T16:37:00.7966538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7966960Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7967364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7967761Z layer_outputs = layer_module( 2025-11-03T16:37:00.7968167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7968610Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7969065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.7969572Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.7970063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.7970511Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.7970657Z 2025-11-03T16:37:00.7970759Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7971155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7971501Z res = mod(**inputs) 2025-11-03T16:37:00.7971908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7972350Z outputs = self.mobilebert( 2025-11-03T16:37:00.7972785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7973234Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7973663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7974103Z layer_outputs = layer_module( 2025-11-03T16:37:00.7974547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.7975013Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.7975470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.7975982Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.7976469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.7976965Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.7977465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.7977940Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.7978103Z 2025-11-03T16:37:00.7978217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7978596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7978942Z res = mod(**inputs) 2025-11-03T16:37:00.7979355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7979814Z outputs = self.mobilebert( 2025-11-03T16:37:00.7980243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7980692Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7981133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7981581Z layer_outputs = layer_module( 2025-11-03T16:37:00.7982011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.7982501Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.7982991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.7983450Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.7983600Z 2025-11-03T16:37:00.7983716Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7984093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7984429Z res = mod(**inputs) 2025-11-03T16:37:00.7984858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7985299Z outputs = self.mobilebert( 2025-11-03T16:37:00.7985727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7986174Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7986609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7987049Z layer_outputs = layer_module( 2025-11-03T16:37:00.7987476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.7987965Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.7988456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.7988940Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.7989124Z 2025-11-03T16:37:00.7989234Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7989614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7989958Z res = mod(**inputs) 2025-11-03T16:37:00.7990360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7990818Z outputs = self.mobilebert( 2025-11-03T16:37:00.7991241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7991685Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7992145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7992571Z layer_outputs = layer_module( 2025-11-03T16:37:00.7992997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.7993528Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.7994182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.7994705Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.7994876Z 2025-11-03T16:37:00.7994991Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.7995390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.7995737Z res = mod(**inputs) 2025-11-03T16:37:00.7996158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.7996600Z outputs = self.mobilebert( 2025-11-03T16:37:00.7997040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.7997496Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.7997939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.7998378Z layer_outputs = layer_module( 2025-11-03T16:37:00.7998802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.7999336Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.7999909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8000407Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8000903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8001372Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8001535Z 2025-11-03T16:37:00.8001663Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8002047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8002393Z res = mod(**inputs) 2025-11-03T16:37:00.8002814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8003254Z outputs = self.mobilebert( 2025-11-03T16:37:00.8003693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8004133Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8004564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8004999Z layer_outputs = layer_module( 2025-11-03T16:37:00.8005423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8005922Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8006443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8006911Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8007378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8007802Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8007958Z 2025-11-03T16:37:00.8008064Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8008438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8008784Z res = mod(**inputs) 2025-11-03T16:37:00.8009193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8009632Z outputs = self.mobilebert( 2025-11-03T16:37:00.8010078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8010520Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8010958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8011389Z layer_outputs = layer_module( 2025-11-03T16:37:00.8011808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8012314Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8012832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8013468Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8013971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8014440Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8014958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8015395Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8015544Z 2025-11-03T16:37:00.8015655Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8016005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8016330Z res = mod(**inputs) 2025-11-03T16:37:00.8016748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8017179Z outputs = self.mobilebert( 2025-11-03T16:37:00.8017613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8018022Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8018440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8018852Z layer_outputs = layer_module( 2025-11-03T16:37:00.8019261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8019760Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8020268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8020722Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8021182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8021633Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8021770Z 2025-11-03T16:37:00.8021872Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8022227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8022550Z res = mod(**inputs) 2025-11-03T16:37:00.8022933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8023345Z outputs = self.mobilebert( 2025-11-03T16:37:00.8023742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8024159Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8024591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8025125Z layer_outputs = layer_module( 2025-11-03T16:37:00.8025554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8026008Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8026462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8026898Z self_outputs = self.self( 2025-11-03T16:37:00.8027323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8027741Z self.value(value_tensor) 2025-11-03T16:37:00.8027863Z 2025-11-03T16:37:00.8027967Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8028325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8028653Z res = mod(**inputs) 2025-11-03T16:37:00.8029069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8029509Z outputs = self.mobilebert( 2025-11-03T16:37:00.8029969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8030419Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8030851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8031295Z layer_outputs = layer_module( 2025-11-03T16:37:00.8031743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8032293Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8032842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8033332Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8033831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8034358Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8034514Z 2025-11-03T16:37:00.8034625Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8035017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8035367Z res = mod(**inputs) 2025-11-03T16:37:00.8035754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8036194Z outputs = self.mobilebert( 2025-11-03T16:37:00.8036597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8037019Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8037435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8037846Z layer_outputs = layer_module( 2025-11-03T16:37:00.8038255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8038759Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8039272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8039726Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8040184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8040621Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8041045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8041481Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8041630Z 2025-11-03T16:37:00.8041746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8042100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8042423Z res = mod(**inputs) 2025-11-03T16:37:00.8042814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8043228Z outputs = self.mobilebert( 2025-11-03T16:37:00.8043631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8044039Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8044470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8044885Z layer_outputs = layer_module( 2025-11-03T16:37:00.8045293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8045716Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8046161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8046581Z self_outputs = self.self( 2025-11-03T16:37:00.8046989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8047413Z self.query(query_tensor) 2025-11-03T16:37:00.8047528Z 2025-11-03T16:37:00.8047629Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8047994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8048320Z res = mod(**inputs) 2025-11-03T16:37:00.8048712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8049127Z outputs = self.mobilebert( 2025-11-03T16:37:00.8049531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8049955Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8050365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8050795Z layer_outputs = layer_module( 2025-11-03T16:37:00.8051200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8051631Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8052083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8052518Z self_outputs = self.self( 2025-11-03T16:37:00.8052942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8053369Z self.key(key_tensor) 2025-11-03T16:37:00.8053491Z 2025-11-03T16:37:00.8053581Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8053812Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8054066Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8054505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8054844Z res = mod(**inputs) 2025-11-03T16:37:00.8055235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8055650Z outputs = self.mobilebert( 2025-11-03T16:37:00.8056053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8056469Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8056905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8057346Z layer_outputs = layer_module( 2025-11-03T16:37:00.8057780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8058234Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8058688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8059178Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8059660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8060096Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8060247Z 2025-11-03T16:37:00.8060365Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8060754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8061109Z res = mod(**inputs) 2025-11-03T16:37:00.8061535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8061978Z outputs = self.mobilebert( 2025-11-03T16:37:00.8062412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8062856Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8063301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8063747Z layer_outputs = layer_module( 2025-11-03T16:37:00.8064179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8064622Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8065079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8065572Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8066072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8066581Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8067080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8067544Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8067708Z 2025-11-03T16:37:00.8067816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8068197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8068538Z res = mod(**inputs) 2025-11-03T16:37:00.8068941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8069395Z outputs = self.mobilebert( 2025-11-03T16:37:00.8069817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8070259Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8070696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8071125Z layer_outputs = layer_module( 2025-11-03T16:37:00.8071550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8072010Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8072473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8072966Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8073454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8073921Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8074162Z 2025-11-03T16:37:00.8074277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8074693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8075046Z res = mod(**inputs) 2025-11-03T16:37:00.8075476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8075927Z outputs = self.mobilebert( 2025-11-03T16:37:00.8076392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8076830Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8077257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8077707Z layer_outputs = layer_module( 2025-11-03T16:37:00.8078149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8078620Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8079097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8079576Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8080083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8080577Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8080759Z 2025-11-03T16:37:00.8080883Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8081311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8081655Z res = mod(**inputs) 2025-11-03T16:37:00.8082090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8082545Z outputs = self.mobilebert( 2025-11-03T16:37:00.8082986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8083453Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8083902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8084372Z layer_outputs = layer_module( 2025-11-03T16:37:00.8084818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8085318Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8085796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8086383Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8086902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8087372Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8087525Z 2025-11-03T16:37:00.8087643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8088024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8088379Z res = mod(**inputs) 2025-11-03T16:37:00.8088797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8089252Z outputs = self.mobilebert( 2025-11-03T16:37:00.8089690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8090131Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8090598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8091056Z layer_outputs = layer_module( 2025-11-03T16:37:00.8091500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8091973Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8092459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8092970Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8093475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8093981Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8094491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8094966Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8095140Z 2025-11-03T16:37:00.8095252Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8095629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8095973Z res = mod(**inputs) 2025-11-03T16:37:00.8096375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8096836Z outputs = self.mobilebert( 2025-11-03T16:37:00.8097271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8097719Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8098159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8098593Z layer_outputs = layer_module( 2025-11-03T16:37:00.8099033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8099501Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8099972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8100459Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8100950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8101396Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8101552Z 2025-11-03T16:37:00.8101661Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8102045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8102383Z res = mod(**inputs) 2025-11-03T16:37:00.8102796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8103208Z outputs = self.mobilebert( 2025-11-03T16:37:00.8103607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8104018Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8104415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8104824Z layer_outputs = layer_module( 2025-11-03T16:37:00.8105227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8105692Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8106130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8106571Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8107059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8107541Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8107721Z 2025-11-03T16:37:00.8107839Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8108232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8108575Z res = mod(**inputs) 2025-11-03T16:37:00.8109001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8109426Z outputs = self.mobilebert( 2025-11-03T16:37:00.8109861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8110306Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8110743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8111189Z layer_outputs = layer_module( 2025-11-03T16:37:00.8111629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8112124Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8112575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8113073Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8113717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8114262Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8114419Z 2025-11-03T16:37:00.8114541Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8114935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8115291Z res = mod(**inputs) 2025-11-03T16:37:00.8115716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8116182Z outputs = self.mobilebert( 2025-11-03T16:37:00.8116572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8116968Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8117365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8117774Z layer_outputs = layer_module( 2025-11-03T16:37:00.8118175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8118620Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8119051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8119519Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8119982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8120442Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8120928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8121355Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8121512Z 2025-11-03T16:37:00.8121616Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8122002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8122328Z res = mod(**inputs) 2025-11-03T16:37:00.8122710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8123129Z outputs = self.mobilebert( 2025-11-03T16:37:00.8123533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8123957Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8124369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8124774Z layer_outputs = layer_module( 2025-11-03T16:37:00.8125179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8125617Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8126052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8126501Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8126979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8127411Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8127556Z 2025-11-03T16:37:00.8127661Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8128019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8128342Z res = mod(**inputs) 2025-11-03T16:37:00.8128722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8130516Z outputs = self.mobilebert( 2025-11-03T16:37:00.8130921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8131364Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8131777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8132253Z layer_outputs = layer_module( 2025-11-03T16:37:00.8132658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8133118Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8133587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8134037Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8134484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8134941Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8135116Z 2025-11-03T16:37:00.8135219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8135575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8135891Z res = mod(**inputs) 2025-11-03T16:37:00.8136299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8136710Z outputs = self.mobilebert( 2025-11-03T16:37:00.8137113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8137526Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8137945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8138406Z layer_outputs = layer_module( 2025-11-03T16:37:00.8138823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8139269Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8139704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8140164Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8140643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8141073Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8141211Z 2025-11-03T16:37:00.8141319Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8141675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8141990Z res = mod(**inputs) 2025-11-03T16:37:00.8142376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8142825Z outputs = self.mobilebert( 2025-11-03T16:37:00.8143252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8143688Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8144138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8144571Z layer_outputs = layer_module( 2025-11-03T16:37:00.8145010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8145468Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8145917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8146404Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8146874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8147367Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8147862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8148318Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8148482Z 2025-11-03T16:37:00.8148592Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8148971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8149316Z res = mod(**inputs) 2025-11-03T16:37:00.8149735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8150167Z outputs = self.mobilebert( 2025-11-03T16:37:00.8150600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8151051Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8151501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8151945Z layer_outputs = layer_module( 2025-11-03T16:37:00.8152364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8152867Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8153360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8153815Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8153964Z 2025-11-03T16:37:00.8154169Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8154552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8154902Z res = mod(**inputs) 2025-11-03T16:37:00.8155321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8155765Z outputs = self.mobilebert( 2025-11-03T16:37:00.8156186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8156631Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8157069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8157536Z layer_outputs = layer_module( 2025-11-03T16:37:00.8157969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8158449Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8158940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8159417Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8159591Z 2025-11-03T16:37:00.8159710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8160107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8160444Z res = mod(**inputs) 2025-11-03T16:37:00.8160857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8161311Z outputs = self.mobilebert( 2025-11-03T16:37:00.8161733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8162159Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8162590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8163010Z layer_outputs = layer_module( 2025-11-03T16:37:00.8163403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8163889Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8164376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8164800Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8164958Z 2025-11-03T16:37:00.8165061Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8165405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8165720Z res = mod(**inputs) 2025-11-03T16:37:00.8166109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8166509Z outputs = self.mobilebert( 2025-11-03T16:37:00.8166899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8167300Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8167727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8168117Z layer_outputs = layer_module( 2025-11-03T16:37:00.8168513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8168998Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8169486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8169942Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8170400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8170834Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8170989Z 2025-11-03T16:37:00.8171093Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8171453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8171786Z res = mod(**inputs) 2025-11-03T16:37:00.8172176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8172590Z outputs = self.mobilebert( 2025-11-03T16:37:00.8172995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8173413Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8173820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8174240Z layer_outputs = layer_module( 2025-11-03T16:37:00.8174639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8175126Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8175635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8176084Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8176544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8176963Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8177099Z 2025-11-03T16:37:00.8177208Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8177558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8177868Z res = mod(**inputs) 2025-11-03T16:37:00.8178250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8178654Z outputs = self.mobilebert( 2025-11-03T16:37:00.8179053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8179463Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8179870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8180302Z layer_outputs = layer_module( 2025-11-03T16:37:00.8180717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8181225Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8181739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8182206Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8182674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8183135Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8183600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8184026Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8184184Z 2025-11-03T16:37:00.8184287Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8184646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8184974Z res = mod(**inputs) 2025-11-03T16:37:00.8185367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8185773Z outputs = self.mobilebert( 2025-11-03T16:37:00.8186175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8186606Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8187009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8187420Z layer_outputs = layer_module( 2025-11-03T16:37:00.8187820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8188398Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8188901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8189352Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8189815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8190230Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8190378Z 2025-11-03T16:37:00.8190478Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8190838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8191163Z res = mod(**inputs) 2025-11-03T16:37:00.8191545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8192004Z outputs = self.mobilebert( 2025-11-03T16:37:00.8192400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8192806Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8193204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8193598Z layer_outputs = layer_module( 2025-11-03T16:37:00.8194123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8194599Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8195091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8195497Z self_outputs = self.self( 2025-11-03T16:37:00.8195880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8196295Z self.value(value_tensor) 2025-11-03T16:37:00.8196437Z 2025-11-03T16:37:00.8196539Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8196895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8197207Z res = mod(**inputs) 2025-11-03T16:37:00.8197586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8197986Z outputs = self.mobilebert( 2025-11-03T16:37:00.8198379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8198782Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8199174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8199574Z layer_outputs = layer_module( 2025-11-03T16:37:00.8199969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8200456Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8200964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8201395Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8201828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8202235Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8202366Z 2025-11-03T16:37:00.8202474Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8202822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8203128Z res = mod(**inputs) 2025-11-03T16:37:00.8203500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8203923Z outputs = self.mobilebert( 2025-11-03T16:37:00.8204323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8204728Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8205139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8205545Z layer_outputs = layer_module( 2025-11-03T16:37:00.8205948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8206440Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8206936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8207383Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8207833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8208262Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8208711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8209112Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8209258Z 2025-11-03T16:37:00.8209357Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8209701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8210019Z res = mod(**inputs) 2025-11-03T16:37:00.8210411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8210813Z outputs = self.mobilebert( 2025-11-03T16:37:00.8211194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8211591Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8211981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8212378Z layer_outputs = layer_module( 2025-11-03T16:37:00.8212763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8213175Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8213772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8214189Z self_outputs = self.self( 2025-11-03T16:37:00.8214574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8215009Z self.query(query_tensor) 2025-11-03T16:37:00.8215125Z 2025-11-03T16:37:00.8215223Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8215561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8215867Z res = mod(**inputs) 2025-11-03T16:37:00.8216224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8216612Z outputs = self.mobilebert( 2025-11-03T16:37:00.8216990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8217385Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8217772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8218179Z layer_outputs = layer_module( 2025-11-03T16:37:00.8218564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8218965Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8219366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8219745Z self_outputs = self.self( 2025-11-03T16:37:00.8220123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8220510Z self.key(key_tensor) 2025-11-03T16:37:00.8220612Z 2025-11-03T16:37:00.8220699Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8220904Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8221121Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8221467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8221769Z res = mod(**inputs) 2025-11-03T16:37:00.8222135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8222551Z outputs = self.mobilebert( 2025-11-03T16:37:00.8222934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8223323Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8223708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8224116Z layer_outputs = layer_module( 2025-11-03T16:37:00.8224495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8224903Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8225309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8225752Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8226203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8226604Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8226747Z 2025-11-03T16:37:00.8226844Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8227184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8227496Z res = mod(**inputs) 2025-11-03T16:37:00.8227861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8228267Z outputs = self.mobilebert( 2025-11-03T16:37:00.8228652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8229048Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8229438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8229820Z layer_outputs = layer_module( 2025-11-03T16:37:00.8230204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8230607Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8231012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8231453Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8231901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8232346Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8232792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8233208Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8233349Z 2025-11-03T16:37:00.8233454Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8233793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8234214Z res = mod(**inputs) 2025-11-03T16:37:00.8234615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8235037Z outputs = self.mobilebert( 2025-11-03T16:37:00.8235445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8235900Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8236328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8236746Z layer_outputs = layer_module( 2025-11-03T16:37:00.8237156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8237580Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8238038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8238493Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8238948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8239375Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8239516Z 2025-11-03T16:37:00.8239618Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8239974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8240296Z res = mod(**inputs) 2025-11-03T16:37:00.8240683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8241096Z outputs = self.mobilebert( 2025-11-03T16:37:00.8241495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8241911Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8242326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8242758Z layer_outputs = layer_module( 2025-11-03T16:37:00.8243167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8243602Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8244040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8244498Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8244958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8245388Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8245546Z 2025-11-03T16:37:00.8245644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8246004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8246311Z res = mod(**inputs) 2025-11-03T16:37:00.8246680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8247067Z outputs = self.mobilebert( 2025-11-03T16:37:00.8247448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8247842Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8248228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8248616Z layer_outputs = layer_module( 2025-11-03T16:37:00.8248991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8249407Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8249821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8250265Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8250721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8251124Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8251264Z 2025-11-03T16:37:00.8251362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8251713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8252022Z res = mod(**inputs) 2025-11-03T16:37:00.8252386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8252776Z outputs = self.mobilebert( 2025-11-03T16:37:00.8253157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8253545Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8253933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8254312Z layer_outputs = layer_module( 2025-11-03T16:37:00.8254695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8255108Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8255522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8255962Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8256421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8256866Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8257309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8257724Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8257865Z 2025-11-03T16:37:00.8257972Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8258162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8258224Z res = mod(**inputs) 2025-11-03T16:37:00.8258502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8258592Z outputs = self.mobilebert( 2025-11-03T16:37:00.8258864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8258931Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8259204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8259271Z layer_outputs = layer_module( 2025-11-03T16:37:00.8259536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8259634Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8259899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8260016Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8260283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8260362Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8260373Z 2025-11-03T16:37:00.8260470Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8260673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8260744Z res = mod(**inputs) 2025-11-03T16:37:00.8261024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8261097Z outputs = self.mobilebert( 2025-11-03T16:37:00.8261409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8261481Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8261760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8261827Z layer_outputs = layer_module( 2025-11-03T16:37:00.8262099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8262191Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8262460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8262576Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8262846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8262962Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8262967Z 2025-11-03T16:37:00.8263081Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8263279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8263342Z res = mod(**inputs) 2025-11-03T16:37:00.8263613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8263692Z outputs = self.mobilebert( 2025-11-03T16:37:00.8263963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8264041Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8264312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8264378Z layer_outputs = layer_module( 2025-11-03T16:37:00.8264655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8264773Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8265044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8265165Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8265439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8265519Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8265522Z 2025-11-03T16:37:00.8265618Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8265824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8265886Z res = mod(**inputs) 2025-11-03T16:37:00.8266154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8266223Z outputs = self.mobilebert( 2025-11-03T16:37:00.8266492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8266567Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8266842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8266918Z layer_outputs = layer_module( 2025-11-03T16:37:00.8267188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8267299Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8267568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8267689Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8267961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8268076Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8268349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8268436Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8268439Z 2025-11-03T16:37:00.8268543Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8268730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8268791Z res = mod(**inputs) 2025-11-03T16:37:00.8269061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8269145Z outputs = self.mobilebert( 2025-11-03T16:37:00.8269418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8269485Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8269751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8269825Z layer_outputs = layer_module( 2025-11-03T16:37:00.8270091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8270184Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8270451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8270579Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8270845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8270924Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8270927Z 2025-11-03T16:37:00.8271032Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8271220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8271288Z res = mod(**inputs) 2025-11-03T16:37:00.8271550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8271616Z outputs = self.mobilebert( 2025-11-03T16:37:00.8271892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8271961Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8272230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8272295Z layer_outputs = layer_module( 2025-11-03T16:37:00.8272583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8272669Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8272930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8273040Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8273319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8273430Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8273436Z 2025-11-03T16:37:00.8273530Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8273719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8273788Z res = mod(**inputs) 2025-11-03T16:37:00.8274134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8274220Z outputs = self.mobilebert( 2025-11-03T16:37:00.8274516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8274597Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8274883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8274956Z layer_outputs = layer_module( 2025-11-03T16:37:00.8275247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8275372Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8275655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8275778Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8276056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8276149Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8276153Z 2025-11-03T16:37:00.8276256Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8276466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8276534Z res = mod(**inputs) 2025-11-03T16:37:00.8276842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8276914Z outputs = self.mobilebert( 2025-11-03T16:37:00.8277196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8277277Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8277558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8277635Z layer_outputs = layer_module( 2025-11-03T16:37:00.8277915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8278007Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8278295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8278420Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8278704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8278848Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8279136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8279227Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8279231Z 2025-11-03T16:37:00.8279332Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8279553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8279619Z res = mod(**inputs) 2025-11-03T16:37:00.8279905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8279976Z outputs = self.mobilebert( 2025-11-03T16:37:00.8280259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8280335Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8280617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8280695Z layer_outputs = layer_module( 2025-11-03T16:37:00.8280976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8281104Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8281385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8281526Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8281530Z 2025-11-03T16:37:00.8281636Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8281840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8281909Z res = mod(**inputs) 2025-11-03T16:37:00.8282199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8282268Z outputs = self.mobilebert( 2025-11-03T16:37:00.8282566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8282638Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8282929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8283014Z layer_outputs = layer_module( 2025-11-03T16:37:00.8283302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8283422Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8283704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8283825Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8283829Z 2025-11-03T16:37:00.8283931Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8284137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8284204Z res = mod(**inputs) 2025-11-03T16:37:00.8284486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8284569Z outputs = self.mobilebert( 2025-11-03T16:37:00.8284853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8284932Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8285230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8285309Z layer_outputs = layer_module( 2025-11-03T16:37:00.8285590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8285751Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8286052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8286146Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8286153Z 2025-11-03T16:37:00.8286261Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8286462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8286529Z res = mod(**inputs) 2025-11-03T16:37:00.8286793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8286858Z outputs = self.mobilebert( 2025-11-03T16:37:00.8287132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8287197Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8287471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8287537Z layer_outputs = layer_module( 2025-11-03T16:37:00.8287803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8287991Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8288257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8288380Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8288644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8288734Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8288737Z 2025-11-03T16:37:00.8288833Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8289020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8289108Z res = mod(**inputs) 2025-11-03T16:37:00.8289372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8289443Z outputs = self.mobilebert( 2025-11-03T16:37:00.8289714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8289780Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8290058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8290124Z layer_outputs = layer_module( 2025-11-03T16:37:00.8290402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8290552Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8290830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8290948Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8291268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8291357Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8291361Z 2025-11-03T16:37:00.8291457Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8291650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8291710Z res = mod(**inputs) 2025-11-03T16:37:00.8291990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8292064Z outputs = self.mobilebert( 2025-11-03T16:37:00.8292334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8292411Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8292678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8292753Z layer_outputs = layer_module( 2025-11-03T16:37:00.8293019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8293164Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8293441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8293557Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8293834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8293966Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8294240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8294326Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8294330Z 2025-11-03T16:37:00.8294427Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8294619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8294679Z res = mod(**inputs) 2025-11-03T16:37:00.8294958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8295025Z outputs = self.mobilebert( 2025-11-03T16:37:00.8295300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8295375Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8295637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8295711Z layer_outputs = layer_module( 2025-11-03T16:37:00.8295974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8296135Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8296400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8296504Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8296772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8296851Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8296855Z 2025-11-03T16:37:00.8296958Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8297159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8297222Z res = mod(**inputs) 2025-11-03T16:37:00.8297495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8297562Z outputs = self.mobilebert( 2025-11-03T16:37:00.8297850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8297919Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8298193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8298261Z layer_outputs = layer_module( 2025-11-03T16:37:00.8298535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8298625Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8298893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8298970Z self_outputs = self.self( 2025-11-03T16:37:00.8299235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8299304Z self.value(value_tensor) 2025-11-03T16:37:00.8299308Z 2025-11-03T16:37:00.8299412Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8299600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8299685Z res = mod(**inputs) 2025-11-03T16:37:00.8299956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8300031Z outputs = self.mobilebert( 2025-11-03T16:37:00.8300363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8300430Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8300704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8300769Z layer_outputs = layer_module( 2025-11-03T16:37:00.8301046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8301196Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8301479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8301590Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8301858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8301942Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8301945Z 2025-11-03T16:37:00.8302040Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8302232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8302293Z res = mod(**inputs) 2025-11-03T16:37:00.8302569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8302642Z outputs = self.mobilebert( 2025-11-03T16:37:00.8302903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8302973Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8303244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8303311Z layer_outputs = layer_module( 2025-11-03T16:37:00.8303577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8303721Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8303999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8304102Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8304368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8304447Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8304701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8304794Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8304798Z 2025-11-03T16:37:00.8304890Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8305076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8305135Z res = mod(**inputs) 2025-11-03T16:37:00.8305394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8305463Z outputs = self.mobilebert( 2025-11-03T16:37:00.8305736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8305809Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8306067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8306139Z layer_outputs = layer_module( 2025-11-03T16:37:00.8306397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8306477Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8306740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8306805Z self_outputs = self.self( 2025-11-03T16:37:00.8307068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8307152Z self.query(query_tensor) 2025-11-03T16:37:00.8307156Z 2025-11-03T16:37:00.8307250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8307437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8307496Z res = mod(**inputs) 2025-11-03T16:37:00.8307760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8307825Z outputs = self.mobilebert( 2025-11-03T16:37:00.8308095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8308164Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8308426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8308504Z layer_outputs = layer_module( 2025-11-03T16:37:00.8308769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8308857Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8309136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8309203Z self_outputs = self.self( 2025-11-03T16:37:00.8309477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8309542Z self.key(key_tensor) 2025-11-03T16:37:00.8309546Z 2025-11-03T16:37:00.8309653Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8309732Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8309838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8310026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8310086Z res = mod(**inputs) 2025-11-03T16:37:00.8310358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8310425Z outputs = self.mobilebert( 2025-11-03T16:37:00.8310698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8310766Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8311031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8311106Z layer_outputs = layer_module( 2025-11-03T16:37:00.8311371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8311475Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8311739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8311859Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8312130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8312210Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8312213Z 2025-11-03T16:37:00.8312316Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8312504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8312573Z res = mod(**inputs) 2025-11-03T16:37:00.8312834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8312917Z outputs = self.mobilebert( 2025-11-03T16:37:00.8313351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8313431Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8313722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8313790Z layer_outputs = layer_module( 2025-11-03T16:37:00.8314099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8314199Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8314470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8314598Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8314873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8315002Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8315314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8315415Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8315418Z 2025-11-03T16:37:00.8315519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8315699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8315765Z res = mod(**inputs) 2025-11-03T16:37:00.8316043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8316110Z outputs = self.mobilebert( 2025-11-03T16:37:00.8316378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8316445Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8316713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8316780Z layer_outputs = layer_module( 2025-11-03T16:37:00.8317050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8317139Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8317397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8317511Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8317771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8317888Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8317892Z 2025-11-03T16:37:00.8317983Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8318172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8318230Z res = mod(**inputs) 2025-11-03T16:37:00.8318490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8318562Z outputs = self.mobilebert( 2025-11-03T16:37:00.8318824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8318897Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8319157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8319261Z layer_outputs = layer_module( 2025-11-03T16:37:00.8319523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8319612Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8319875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8319977Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8320242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8320350Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8320353Z 2025-11-03T16:37:00.8320445Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8320639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8320700Z res = mod(**inputs) 2025-11-03T16:37:00.8320971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8321053Z outputs = self.mobilebert( 2025-11-03T16:37:00.8321324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8321398Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8321659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8321744Z layer_outputs = layer_module( 2025-11-03T16:37:00.8322003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8322099Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8322358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8322473Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8322747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8322825Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8322828Z 2025-11-03T16:37:00.8322929Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8323112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8323171Z res = mod(**inputs) 2025-11-03T16:37:00.8323435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8323514Z outputs = self.mobilebert( 2025-11-03T16:37:00.8323777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8323842Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8324108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8324174Z layer_outputs = layer_module( 2025-11-03T16:37:00.8324430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8324525Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8324784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8324923Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8325179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8325290Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8325559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8325644Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8325647Z 2025-11-03T16:37:00.8325748Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8325930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8325999Z res = mod(**inputs) 2025-11-03T16:37:00.8326255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8326321Z outputs = self.mobilebert( 2025-11-03T16:37:00.8326585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8326651Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8326937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8327006Z layer_outputs = layer_module( 2025-11-03T16:37:00.8327261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8327354Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8327628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8327739Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8327999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8328081Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8328084Z 2025-11-03T16:37:00.8328177Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8328359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8328427Z res = mod(**inputs) 2025-11-03T16:37:00.8328684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8328756Z outputs = self.mobilebert( 2025-11-03T16:37:00.8329016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8329082Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8329363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8329428Z layer_outputs = layer_module( 2025-11-03T16:37:00.8329689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8329777Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8330041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8330143Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8330400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8330512Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8330515Z 2025-11-03T16:37:00.8330627Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8330819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8330879Z res = mod(**inputs) 2025-11-03T16:37:00.8331156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8331230Z outputs = self.mobilebert( 2025-11-03T16:37:00.8331496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8331570Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8331837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8331911Z layer_outputs = layer_module( 2025-11-03T16:37:00.8332177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8332268Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8332541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8332686Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8332961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8333040Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8333043Z 2025-11-03T16:37:00.8333145Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8333342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8333405Z res = mod(**inputs) 2025-11-03T16:37:00.8333673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8333742Z outputs = self.mobilebert( 2025-11-03T16:37:00.8334012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8334078Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8334341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8334415Z layer_outputs = layer_module( 2025-11-03T16:37:00.8334681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8334777Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8335041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8335183Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8335451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8335565Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8335841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8335927Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8335930Z 2025-11-03T16:37:00.8336033Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8336220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8336280Z res = mod(**inputs) 2025-11-03T16:37:00.8336555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8336639Z outputs = self.mobilebert( 2025-11-03T16:37:00.8336907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8336974Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8337248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8337313Z layer_outputs = layer_module( 2025-11-03T16:37:00.8337580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8337677Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8337944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8338057Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8338327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8338406Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8338418Z 2025-11-03T16:37:00.8338528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8338717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8338786Z res = mod(**inputs) 2025-11-03T16:37:00.8339052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8339125Z outputs = self.mobilebert( 2025-11-03T16:37:00.8339406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8339476Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8339747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8339814Z layer_outputs = layer_module( 2025-11-03T16:37:00.8340087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8340174Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8340438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8340549Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8340814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8340925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8340945Z 2025-11-03T16:37:00.8341043Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8341233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8341294Z res = mod(**inputs) 2025-11-03T16:37:00.8341565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8341640Z outputs = self.mobilebert( 2025-11-03T16:37:00.8341910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8341985Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8342260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8342326Z layer_outputs = layer_module( 2025-11-03T16:37:00.8342598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8342700Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8342975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8343093Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8343369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8343448Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8343452Z 2025-11-03T16:37:00.8343548Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8343746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8343806Z res = mod(**inputs) 2025-11-03T16:37:00.8344084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8344150Z outputs = self.mobilebert( 2025-11-03T16:37:00.8344419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8344510Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8344777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8344855Z layer_outputs = layer_module( 2025-11-03T16:37:00.8345120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8345227Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8345494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8345613Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8345886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8346002Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8346274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8346358Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8346361Z 2025-11-03T16:37:00.8346462Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8346649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8346709Z res = mod(**inputs) 2025-11-03T16:37:00.8346984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8347067Z outputs = self.mobilebert( 2025-11-03T16:37:00.8347341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8347409Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8347672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8347747Z layer_outputs = layer_module( 2025-11-03T16:37:00.8348009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8348132Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8348395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8348495Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8348499Z 2025-11-03T16:37:00.8348594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8348780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8348845Z res = mod(**inputs) 2025-11-03T16:37:00.8349110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8349180Z outputs = self.mobilebert( 2025-11-03T16:37:00.8349443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8349509Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8349784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8349852Z layer_outputs = layer_module( 2025-11-03T16:37:00.8350126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8350237Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8350524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8350633Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8350637Z 2025-11-03T16:37:00.8350733Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8350928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8350989Z res = mod(**inputs) 2025-11-03T16:37:00.8351279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8351348Z outputs = self.mobilebert( 2025-11-03T16:37:00.8351614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8351689Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8351954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8352028Z layer_outputs = layer_module( 2025-11-03T16:37:00.8352291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8352447Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8352713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8352802Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8352821Z 2025-11-03T16:37:00.8352927Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8353115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8353181Z res = mod(**inputs) 2025-11-03T16:37:00.8353451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8353519Z outputs = self.mobilebert( 2025-11-03T16:37:00.8353804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8353873Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8354239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8354313Z layer_outputs = layer_module( 2025-11-03T16:37:00.8354630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8354785Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8355061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8355201Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8355466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8355562Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8355565Z 2025-11-03T16:37:00.8355663Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8355851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8355920Z res = mod(**inputs) 2025-11-03T16:37:00.8356187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8356263Z outputs = self.mobilebert( 2025-11-03T16:37:00.8356543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8356622Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8356884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8356948Z layer_outputs = layer_module( 2025-11-03T16:37:00.8357238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8357388Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8357664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8357784Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8358061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8358141Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8358144Z 2025-11-03T16:37:00.8358240Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8358437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8358499Z res = mod(**inputs) 2025-11-03T16:37:00.8358770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8358836Z outputs = self.mobilebert( 2025-11-03T16:37:00.8359100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8359191Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8359457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8359530Z layer_outputs = layer_module( 2025-11-03T16:37:00.8359795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8359949Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8360217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8360334Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8360620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8360737Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8361005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8361092Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8361095Z 2025-11-03T16:37:00.8361191Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8361384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8361444Z res = mod(**inputs) 2025-11-03T16:37:00.8361716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8361783Z outputs = self.mobilebert( 2025-11-03T16:37:00.8362052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8362122Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8362383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8362472Z layer_outputs = layer_module( 2025-11-03T16:37:00.8362737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8362896Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8363177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8363281Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8363550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8363630Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8363634Z 2025-11-03T16:37:00.8363738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8363924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8363990Z res = mod(**inputs) 2025-11-03T16:37:00.8364256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8364321Z outputs = self.mobilebert( 2025-11-03T16:37:00.8364593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8364660Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8364930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8365011Z layer_outputs = layer_module( 2025-11-03T16:37:00.8365274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8365360Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8365624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8365698Z self_outputs = self.self( 2025-11-03T16:37:00.8365960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8366033Z self.value(value_tensor) 2025-11-03T16:37:00.8366038Z 2025-11-03T16:37:00.8366132Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8366316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8367066Z res = mod(**inputs) 2025-11-03T16:37:00.8367330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8367403Z outputs = self.mobilebert( 2025-11-03T16:37:00.8367668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8367736Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8368004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8368070Z layer_outputs = layer_module( 2025-11-03T16:37:00.8368343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8368492Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8368764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8368868Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8369148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8369237Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8369241Z 2025-11-03T16:37:00.8369336Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8369527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8369586Z res = mod(**inputs) 2025-11-03T16:37:00.8369864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8369942Z outputs = self.mobilebert( 2025-11-03T16:37:00.8370208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8370284Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8370552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8370628Z layer_outputs = layer_module( 2025-11-03T16:37:00.8370894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8371043Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8371316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8371417Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8371704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8371786Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8372058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8372142Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8372146Z 2025-11-03T16:37:00.8372242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8372435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8372495Z res = mod(**inputs) 2025-11-03T16:37:00.8372766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8372831Z outputs = self.mobilebert( 2025-11-03T16:37:00.8373108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8373184Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8373447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8373521Z layer_outputs = layer_module( 2025-11-03T16:37:00.8373784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8373870Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8374141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8374211Z self_outputs = self.self( 2025-11-03T16:37:00.8374487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8374559Z self.query(query_tensor) 2025-11-03T16:37:00.8374562Z 2025-11-03T16:37:00.8374667Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8374856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8374934Z res = mod(**inputs) 2025-11-03T16:37:00.8375224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8375291Z outputs = self.mobilebert( 2025-11-03T16:37:00.8375571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8375686Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8375957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8376035Z layer_outputs = layer_module( 2025-11-03T16:37:00.8376303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8376390Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8376659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8376735Z self_outputs = self.self( 2025-11-03T16:37:00.8377003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8377068Z self.key(key_tensor) 2025-11-03T16:37:00.8377072Z 2025-11-03T16:37:00.8377159Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8377236Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8377342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8377578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8377639Z res = mod(**inputs) 2025-11-03T16:37:00.8377918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8377987Z outputs = self.mobilebert( 2025-11-03T16:37:00.8378273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8378345Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8378623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8378701Z layer_outputs = layer_module( 2025-11-03T16:37:00.8378981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8379085Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8379359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8379484Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8379759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8379839Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8379843Z 2025-11-03T16:37:00.8379951Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8380146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8380216Z res = mod(**inputs) 2025-11-03T16:37:00.8380492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8380570Z outputs = self.mobilebert( 2025-11-03T16:37:00.8380845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8380916Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8381215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8381284Z layer_outputs = layer_module( 2025-11-03T16:37:00.8381559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8381639Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8381925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8382054Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8382328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8382453Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8382726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8382821Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8382825Z 2025-11-03T16:37:00.8382923Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8383115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8383185Z res = mod(**inputs) 2025-11-03T16:37:00.8383457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8383531Z outputs = self.mobilebert( 2025-11-03T16:37:00.8383819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8383888Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8384168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8384237Z layer_outputs = layer_module( 2025-11-03T16:37:00.8384516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8384606Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8384886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8384995Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8385280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8385370Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8385374Z 2025-11-03T16:37:00.8385471Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8385671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8385732Z res = mod(**inputs) 2025-11-03T16:37:00.8386004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8386080Z outputs = self.mobilebert( 2025-11-03T16:37:00.8386354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8386429Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8386700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8386778Z layer_outputs = layer_module( 2025-11-03T16:37:00.8387048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8387153Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8387433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8387541Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8387821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8387946Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8387950Z 2025-11-03T16:37:00.8388052Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8388258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8388319Z res = mod(**inputs) 2025-11-03T16:37:00.8388602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8388671Z outputs = self.mobilebert( 2025-11-03T16:37:00.8388949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8389018Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8389295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8389373Z layer_outputs = layer_module( 2025-11-03T16:37:00.8389654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8389765Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8390030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8390148Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8390419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8390500Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8390503Z 2025-11-03T16:37:00.8390606Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8390793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8390859Z res = mod(**inputs) 2025-11-03T16:37:00.8391126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8391210Z outputs = self.mobilebert( 2025-11-03T16:37:00.8391481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8391550Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8391820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8391886Z layer_outputs = layer_module( 2025-11-03T16:37:00.8392148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8392242Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8392509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8392637Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8392901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8393021Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8393299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8393387Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8393390Z 2025-11-03T16:37:00.8393493Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8393681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8393763Z res = mod(**inputs) 2025-11-03T16:37:00.8394104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8394218Z outputs = self.mobilebert( 2025-11-03T16:37:00.8394492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8394562Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8394843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8394911Z layer_outputs = layer_module( 2025-11-03T16:37:00.8395190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8395280Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8395574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8395688Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8395969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8396058Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8396062Z 2025-11-03T16:37:00.8396158Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8396352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8396413Z res = mod(**inputs) 2025-11-03T16:37:00.8396675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8396749Z outputs = self.mobilebert( 2025-11-03T16:37:00.8397014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8397087Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8397365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8397432Z layer_outputs = layer_module( 2025-11-03T16:37:00.8397702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8397791Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8398062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8398165Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8398437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8398543Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8398546Z 2025-11-03T16:37:00.8398646Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8398839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8398901Z res = mod(**inputs) 2025-11-03T16:37:00.8399188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8399255Z outputs = self.mobilebert( 2025-11-03T16:37:00.8399523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8399601Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8399890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8399967Z layer_outputs = layer_module( 2025-11-03T16:37:00.8400235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8400332Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8400603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8400724Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8400997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8401075Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8401078Z 2025-11-03T16:37:00.8401180Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8401371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8401430Z res = mod(**inputs) 2025-11-03T16:37:00.8401705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8401790Z outputs = self.mobilebert( 2025-11-03T16:37:00.8402061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8402127Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8402458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8402526Z layer_outputs = layer_module( 2025-11-03T16:37:00.8402791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8402889Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8403158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8403299Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8403564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8403680Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8403955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8404041Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8404044Z 2025-11-03T16:37:00.8404147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8404336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8404403Z res = mod(**inputs) 2025-11-03T16:37:00.8404667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8404736Z outputs = self.mobilebert( 2025-11-03T16:37:00.8405009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8405075Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8405365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8405432Z layer_outputs = layer_module( 2025-11-03T16:37:00.8405695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8405790Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8406068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8406182Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8406445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8406531Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8406535Z 2025-11-03T16:37:00.8406631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8406816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8406885Z res = mod(**inputs) 2025-11-03T16:37:00.8407149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8407223Z outputs = self.mobilebert( 2025-11-03T16:37:00.8407492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8407560Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8407855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8407921Z layer_outputs = layer_module( 2025-11-03T16:37:00.8408187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8408272Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8408537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8408636Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8408903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8409015Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8409034Z 2025-11-03T16:37:00.8409132Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8409331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8409390Z res = mod(**inputs) 2025-11-03T16:37:00.8409659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8409732Z outputs = self.mobilebert( 2025-11-03T16:37:00.8409995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8410069Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8410333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8410404Z layer_outputs = layer_module( 2025-11-03T16:37:00.8410669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8410758Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8411028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8411160Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8411429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8411508Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8411511Z 2025-11-03T16:37:00.8411613Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8411809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8411870Z res = mod(**inputs) 2025-11-03T16:37:00.8412144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8412212Z outputs = self.mobilebert( 2025-11-03T16:37:00.8412483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8412551Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8412811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8412881Z layer_outputs = layer_module( 2025-11-03T16:37:00.8413142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8413344Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8413620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8413786Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8414051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8414166Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8414440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8414527Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8414530Z 2025-11-03T16:37:00.8414633Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8414820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8414881Z res = mod(**inputs) 2025-11-03T16:37:00.8415158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8415256Z outputs = self.mobilebert( 2025-11-03T16:37:00.8415527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8415594Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8415867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8415933Z layer_outputs = layer_module( 2025-11-03T16:37:00.8416196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8416320Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8416585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8416674Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8416677Z 2025-11-03T16:37:00.8416772Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8416959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8417026Z res = mod(**inputs) 2025-11-03T16:37:00.8417312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8417388Z outputs = self.mobilebert( 2025-11-03T16:37:00.8417651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8417724Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8418007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8418077Z layer_outputs = layer_module( 2025-11-03T16:37:00.8418350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8418462Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8418732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8418838Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8418841Z 2025-11-03T16:37:00.8418937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8419127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8419188Z res = mod(**inputs) 2025-11-03T16:37:00.8419460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8419546Z outputs = self.mobilebert( 2025-11-03T16:37:00.8419815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8419882Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8420149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8420224Z layer_outputs = layer_module( 2025-11-03T16:37:00.8420488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8420644Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8420909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8420996Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8421024Z 2025-11-03T16:37:00.8421122Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8421316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8421385Z res = mod(**inputs) 2025-11-03T16:37:00.8421652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8421726Z outputs = self.mobilebert( 2025-11-03T16:37:00.8421993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8422059Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8422335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8422401Z layer_outputs = layer_module( 2025-11-03T16:37:00.8422673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8422821Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8423114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8423239Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8423507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8423601Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8423604Z 2025-11-03T16:37:00.8423714Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8423908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8423971Z res = mod(**inputs) 2025-11-03T16:37:00.8424248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8424322Z outputs = self.mobilebert( 2025-11-03T16:37:00.8424591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8424665Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8424933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8424999Z layer_outputs = layer_module( 2025-11-03T16:37:00.8425275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8425422Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8425695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8425826Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8426098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8426178Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8426182Z 2025-11-03T16:37:00.8426276Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8426469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8426529Z res = mod(**inputs) 2025-11-03T16:37:00.8426803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8426868Z outputs = self.mobilebert( 2025-11-03T16:37:00.8427160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8427230Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8427496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8427572Z layer_outputs = layer_module( 2025-11-03T16:37:00.8427837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8427988Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8428254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8428367Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8428643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8428757Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8429034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8429135Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8429138Z 2025-11-03T16:37:00.8429242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8429431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8429491Z res = mod(**inputs) 2025-11-03T16:37:00.8429780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8429848Z outputs = self.mobilebert( 2025-11-03T16:37:00.8430122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8430190Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8430454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8430528Z layer_outputs = layer_module( 2025-11-03T16:37:00.8430788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8430947Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8431211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8431322Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8431584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8431679Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8431682Z 2025-11-03T16:37:00.8431787Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8431973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8432043Z res = mod(**inputs) 2025-11-03T16:37:00.8432309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8432383Z outputs = self.mobilebert( 2025-11-03T16:37:00.8432650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8432718Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8432993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8433075Z layer_outputs = layer_module( 2025-11-03T16:37:00.8433345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8433426Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8433692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8433769Z self_outputs = self.self( 2025-11-03T16:37:00.8434094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8434185Z self.value(value_tensor) 2025-11-03T16:37:00.8434191Z 2025-11-03T16:37:00.8434293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8434491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8434555Z res = mod(**inputs) 2025-11-03T16:37:00.8434826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8434901Z outputs = self.mobilebert( 2025-11-03T16:37:00.8435208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8435288Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8435551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8435618Z layer_outputs = layer_module( 2025-11-03T16:37:00.8435907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8436062Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8436338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8436445Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8436715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8436793Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8436796Z 2025-11-03T16:37:00.8436892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8437085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8437147Z res = mod(**inputs) 2025-11-03T16:37:00.8437417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8437483Z outputs = self.mobilebert( 2025-11-03T16:37:00.8437764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8437840Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8438104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8438178Z layer_outputs = layer_module( 2025-11-03T16:37:00.8438439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8438596Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8438860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8438962Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8439252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8439333Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8439603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8439689Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8439693Z 2025-11-03T16:37:00.8439789Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8439979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8440037Z res = mod(**inputs) 2025-11-03T16:37:00.8440308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8440375Z outputs = self.mobilebert( 2025-11-03T16:37:00.8440644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8440709Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8440972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8441064Z layer_outputs = layer_module( 2025-11-03T16:37:00.8441332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8441420Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8441698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8441768Z self_outputs = self.self( 2025-11-03T16:37:00.8442040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8442109Z self.query(query_tensor) 2025-11-03T16:37:00.8442112Z 2025-11-03T16:37:00.8442217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8442406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8442476Z res = mod(**inputs) 2025-11-03T16:37:00.8442737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8442804Z outputs = self.mobilebert( 2025-11-03T16:37:00.8443072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8443140Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8443409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8443491Z layer_outputs = layer_module( 2025-11-03T16:37:00.8443754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8443842Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8444108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8444182Z self_outputs = self.self( 2025-11-03T16:37:00.8444447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8444516Z self.key(key_tensor) 2025-11-03T16:37:00.8444520Z 2025-11-03T16:37:00.8444598Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8444673Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8444777Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8444987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8445057Z res = mod(**inputs) 2025-11-03T16:37:00.8445322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8445389Z outputs = self.mobilebert( 2025-11-03T16:37:00.8445660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8445727Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8445999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8446067Z layer_outputs = layer_module( 2025-11-03T16:37:00.8446331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8446422Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8446689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8446813Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8447093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8447182Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8447186Z 2025-11-03T16:37:00.8447282Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8447475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8447560Z res = mod(**inputs) 2025-11-03T16:37:00.8447830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8447905Z outputs = self.mobilebert( 2025-11-03T16:37:00.8448172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8448239Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8448520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8448585Z layer_outputs = layer_module( 2025-11-03T16:37:00.8448865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8448950Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8449233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8449348Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8449631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8449756Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8450022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8450115Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8450119Z 2025-11-03T16:37:00.8450215Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8450404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8450471Z res = mod(**inputs) 2025-11-03T16:37:00.8450736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8450826Z outputs = self.mobilebert( 2025-11-03T16:37:00.8451094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8451170Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8451435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8451501Z layer_outputs = layer_module( 2025-11-03T16:37:00.8451775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8451864Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8452139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8452246Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8452514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8452601Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8452604Z 2025-11-03T16:37:00.8452699Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8452913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8452974Z res = mod(**inputs) 2025-11-03T16:37:00.8453243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8453309Z outputs = self.mobilebert( 2025-11-03T16:37:00.8453585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8453663Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8453934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8454010Z layer_outputs = layer_module( 2025-11-03T16:37:00.8454290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8454381Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8454652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8454759Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8455037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8455142Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8455146Z 2025-11-03T16:37:00.8455246Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8455441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8455499Z res = mod(**inputs) 2025-11-03T16:37:00.8455767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8455832Z outputs = self.mobilebert( 2025-11-03T16:37:00.8456094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8456159Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8456415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8456486Z layer_outputs = layer_module( 2025-11-03T16:37:00.8456745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8456856Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8457112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8457232Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8457492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8457568Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8457571Z 2025-11-03T16:37:00.8457669Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8457849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8457916Z res = mod(**inputs) 2025-11-03T16:37:00.8458172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8458248Z outputs = self.mobilebert( 2025-11-03T16:37:00.8458504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8458571Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8458854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8458920Z layer_outputs = layer_module( 2025-11-03T16:37:00.8459180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8459266Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8459535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8459658Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8459916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8460032Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8460289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8460379Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8460382Z 2025-11-03T16:37:00.8460477Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8460664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8460734Z res = mod(**inputs) 2025-11-03T16:37:00.8460991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8461084Z outputs = self.mobilebert( 2025-11-03T16:37:00.8461344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8461412Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8461685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8461752Z layer_outputs = layer_module( 2025-11-03T16:37:00.8462026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8462114Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8462389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8462494Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8462781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8462868Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8462872Z 2025-11-03T16:37:00.8462968Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8463164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8463226Z res = mod(**inputs) 2025-11-03T16:37:00.8463490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8463563Z outputs = self.mobilebert( 2025-11-03T16:37:00.8463829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8463905Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8464173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8464247Z layer_outputs = layer_module( 2025-11-03T16:37:00.8464514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8464618Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8464890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8464993Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8465280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8465388Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8465392Z 2025-11-03T16:37:00.8465492Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8465690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8465750Z res = mod(**inputs) 2025-11-03T16:37:00.8466022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8466091Z outputs = self.mobilebert( 2025-11-03T16:37:00.8466363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8466429Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8466696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8466771Z layer_outputs = layer_module( 2025-11-03T16:37:00.8467036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8467160Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8467426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8467545Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8467818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8467898Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8467901Z 2025-11-03T16:37:00.8468005Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8468195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8468262Z res = mod(**inputs) 2025-11-03T16:37:00.8468529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8468614Z outputs = self.mobilebert( 2025-11-03T16:37:00.8468887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8468957Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8469230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8469296Z layer_outputs = layer_module( 2025-11-03T16:37:00.8469561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8469656Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8469924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8470052Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8470314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8470436Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8470753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8470840Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8470844Z 2025-11-03T16:37:00.8470948Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8471132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8471212Z res = mod(**inputs) 2025-11-03T16:37:00.8471477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8471554Z outputs = self.mobilebert( 2025-11-03T16:37:00.8471823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8471890Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8472164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8472229Z layer_outputs = layer_module( 2025-11-03T16:37:00.8472500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8472592Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8472861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8472975Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8473251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8473339Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8473342Z 2025-11-03T16:37:00.8473436Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8473629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8473688Z res = mod(**inputs) 2025-11-03T16:37:00.8473961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8474107Z outputs = self.mobilebert( 2025-11-03T16:37:00.8474391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8474467Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8474762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8474831Z layer_outputs = layer_module( 2025-11-03T16:37:00.8475121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8475213Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8475498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8475605Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8475887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8475999Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8476004Z 2025-11-03T16:37:00.8476105Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8476309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8476372Z res = mod(**inputs) 2025-11-03T16:37:00.8476668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8476739Z outputs = self.mobilebert( 2025-11-03T16:37:00.8477011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8477090Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8477377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8477453Z layer_outputs = layer_module( 2025-11-03T16:37:00.8477722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8477821Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8478091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8478211Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8478489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8478569Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8478573Z 2025-11-03T16:37:00.8478677Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8478869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8478931Z res = mod(**inputs) 2025-11-03T16:37:00.8479209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8479295Z outputs = self.mobilebert( 2025-11-03T16:37:00.8479577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8479647Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8479924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8479991Z layer_outputs = layer_module( 2025-11-03T16:37:00.8480261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8480361Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8480638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8480782Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8481050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8481166Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8481445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8481532Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8481536Z 2025-11-03T16:37:00.8481643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8481832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8481900Z res = mod(**inputs) 2025-11-03T16:37:00.8482172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8482242Z outputs = self.mobilebert( 2025-11-03T16:37:00.8482522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8482590Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8482886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8482955Z layer_outputs = layer_module( 2025-11-03T16:37:00.8483227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8483367Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8483648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8483737Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8483740Z 2025-11-03T16:37:00.8483836Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8484031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8484093Z res = mod(**inputs) 2025-11-03T16:37:00.8484363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8484437Z outputs = self.mobilebert( 2025-11-03T16:37:00.8484701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8484776Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8485043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8485110Z layer_outputs = layer_module( 2025-11-03T16:37:00.8485404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8485515Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8485791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8485895Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8485898Z 2025-11-03T16:37:00.8485999Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8486184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8486243Z res = mod(**inputs) 2025-11-03T16:37:00.8486518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8486597Z outputs = self.mobilebert( 2025-11-03T16:37:00.8486883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8486950Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8487226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8487300Z layer_outputs = layer_module( 2025-11-03T16:37:00.8487575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8487732Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8488011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8488108Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8488114Z 2025-11-03T16:37:00.8488213Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8488404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8488475Z res = mod(**inputs) 2025-11-03T16:37:00.8488766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8488840Z outputs = self.mobilebert( 2025-11-03T16:37:00.8489096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8489162Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8489449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8489515Z layer_outputs = layer_module( 2025-11-03T16:37:00.8489795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8489940Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8490205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8490319Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8490578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8490670Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8490673Z 2025-11-03T16:37:00.8490769Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8490954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8491016Z res = mod(**inputs) 2025-11-03T16:37:00.8491297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8491361Z outputs = self.mobilebert( 2025-11-03T16:37:00.8491623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8491696Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8491954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8492029Z layer_outputs = layer_module( 2025-11-03T16:37:00.8492291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8492434Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8492713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8492827Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8493088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8493167Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8493170Z 2025-11-03T16:37:00.8493270Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8493454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8493511Z res = mod(**inputs) 2025-11-03T16:37:00.8493776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8493841Z outputs = self.mobilebert( 2025-11-03T16:37:00.8494105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8494172Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8494427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8494513Z layer_outputs = layer_module( 2025-11-03T16:37:00.8494771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8494919Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8495194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8495316Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8495573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8495685Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8495947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8496032Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8496036Z 2025-11-03T16:37:00.8496136Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8496315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8496381Z res = mod(**inputs) 2025-11-03T16:37:00.8496635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8496701Z outputs = self.mobilebert( 2025-11-03T16:37:00.8496962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8497045Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8497316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8497380Z layer_outputs = layer_module( 2025-11-03T16:37:00.8497647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8497803Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8498070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8498180Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8498451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8498550Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8498554Z 2025-11-03T16:37:00.8498647Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8498830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8498896Z res = mod(**inputs) 2025-11-03T16:37:00.8499157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8499231Z outputs = self.mobilebert( 2025-11-03T16:37:00.8499490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8499556Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8499821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8499888Z layer_outputs = layer_module( 2025-11-03T16:37:00.8500152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8500231Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8500507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8500574Z self_outputs = self.self( 2025-11-03T16:37:00.8500831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8500903Z self.value(value_tensor) 2025-11-03T16:37:00.8500926Z 2025-11-03T16:37:00.8501022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8501211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8501272Z res = mod(**inputs) 2025-11-03T16:37:00.8501529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8501601Z outputs = self.mobilebert( 2025-11-03T16:37:00.8501858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8501930Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8502186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8502257Z layer_outputs = layer_module( 2025-11-03T16:37:00.8502516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8502661Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8502946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8503046Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8503315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8503390Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8503393Z 2025-11-03T16:37:00.8503486Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8503675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8503736Z res = mod(**inputs) 2025-11-03T16:37:00.8504001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8504079Z outputs = self.mobilebert( 2025-11-03T16:37:00.8504345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8504411Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8504689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8504762Z layer_outputs = layer_module( 2025-11-03T16:37:00.8505029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8505185Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8505452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8505555Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8505839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8505917Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8506199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8506284Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8506288Z 2025-11-03T16:37:00.8506386Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8506570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8506628Z res = mod(**inputs) 2025-11-03T16:37:00.8506921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8506987Z outputs = self.mobilebert( 2025-11-03T16:37:00.8507261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8507328Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8507590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8507664Z layer_outputs = layer_module( 2025-11-03T16:37:00.8507920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8508008Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8508269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8508344Z self_outputs = self.self( 2025-11-03T16:37:00.8508608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8508693Z self.query(query_tensor) 2025-11-03T16:37:00.8508697Z 2025-11-03T16:37:00.8508802Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8508989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8509058Z res = mod(**inputs) 2025-11-03T16:37:00.8509332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8509405Z outputs = self.mobilebert( 2025-11-03T16:37:00.8509679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8509747Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8510018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8510102Z layer_outputs = layer_module( 2025-11-03T16:37:00.8510377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8510456Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8510726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8510799Z self_outputs = self.self( 2025-11-03T16:37:00.8511069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8511142Z self.key(key_tensor) 2025-11-03T16:37:00.8511146Z 2025-11-03T16:37:00.8511223Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8511298Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8511402Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8511590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8511657Z res = mod(**inputs) 2025-11-03T16:37:00.8511927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8511993Z outputs = self.mobilebert( 2025-11-03T16:37:00.8512299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8512367Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8512647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8512732Z layer_outputs = layer_module( 2025-11-03T16:37:00.8513003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8513085Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8513494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8513628Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8513898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8513985Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8513988Z 2025-11-03T16:37:00.8514135Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8514344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8514409Z res = mod(**inputs) 2025-11-03T16:37:00.8514681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8514802Z outputs = self.mobilebert( 2025-11-03T16:37:00.8515088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8515165Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8515457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8515523Z layer_outputs = layer_module( 2025-11-03T16:37:00.8515799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8515878Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8516207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8516321Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8516611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8516739Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8517008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8517104Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8517107Z 2025-11-03T16:37:00.8517202Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8517397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8517457Z res = mod(**inputs) 2025-11-03T16:37:00.8517722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8517796Z outputs = self.mobilebert( 2025-11-03T16:37:00.8518066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8518139Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8518494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8518562Z layer_outputs = layer_module( 2025-11-03T16:37:00.8518837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8518927Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8519221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8519325Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8519596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8519677Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8519680Z 2025-11-03T16:37:00.8519774Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8519969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8520028Z res = mod(**inputs) 2025-11-03T16:37:00.8520300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8520365Z outputs = self.mobilebert( 2025-11-03T16:37:00.8520639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8520706Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8520976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8521067Z layer_outputs = layer_module( 2025-11-03T16:37:00.8521332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8521430Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8521695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8521800Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8522075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8522184Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8522187Z 2025-11-03T16:37:00.8522291Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8522491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8522558Z res = mod(**inputs) 2025-11-03T16:37:00.8522824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8522892Z outputs = self.mobilebert( 2025-11-03T16:37:00.8523162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8523229Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8523500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8523567Z layer_outputs = layer_module( 2025-11-03T16:37:00.8523835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8523937Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8524202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8524327Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8524609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8524696Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8524699Z 2025-11-03T16:37:00.8524796Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8524982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8525068Z res = mod(**inputs) 2025-11-03T16:37:00.8525332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8525407Z outputs = self.mobilebert( 2025-11-03T16:37:00.8525671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8525738Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8526010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8526077Z layer_outputs = layer_module( 2025-11-03T16:37:00.8526346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8526434Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8526710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8526829Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8527116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8527240Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8527504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8527609Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8527612Z 2025-11-03T16:37:00.8527705Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8527885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8527953Z res = mod(**inputs) 2025-11-03T16:37:00.8528209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8528297Z outputs = self.mobilebert( 2025-11-03T16:37:00.8528553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8528627Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8528896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8528964Z layer_outputs = layer_module( 2025-11-03T16:37:00.8529227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8529312Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8529576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8529678Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8529937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8530020Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8530023Z 2025-11-03T16:37:00.8530116Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8530316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8530377Z res = mod(**inputs) 2025-11-03T16:37:00.8530639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8530703Z outputs = self.mobilebert( 2025-11-03T16:37:00.8530974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8531049Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8531311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8531383Z layer_outputs = layer_module( 2025-11-03T16:37:00.8531638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8531727Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8531990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8532092Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8532358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8532461Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8532464Z 2025-11-03T16:37:00.8532563Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8532761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8532822Z res = mod(**inputs) 2025-11-03T16:37:00.8533095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8533163Z outputs = self.mobilebert( 2025-11-03T16:37:00.8533439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8533503Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8533761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8533833Z layer_outputs = layer_module( 2025-11-03T16:37:00.8534091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8534207Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8534468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8534592Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8534854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8534932Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8534935Z 2025-11-03T16:37:00.8535037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8535221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8535285Z res = mod(**inputs) 2025-11-03T16:37:00.8535547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8535622Z outputs = self.mobilebert( 2025-11-03T16:37:00.8535887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8535954Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8536246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8536315Z layer_outputs = layer_module( 2025-11-03T16:37:00.8536587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8536675Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8536957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8537086Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8537353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8537475Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8537744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8537836Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8537840Z 2025-11-03T16:37:00.8537937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8538123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8538193Z res = mod(**inputs) 2025-11-03T16:37:00.8538459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8538550Z outputs = self.mobilebert( 2025-11-03T16:37:00.8538815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8538883Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8539158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8539227Z layer_outputs = layer_module( 2025-11-03T16:37:00.8539498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8539587Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8539858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8539964Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8540242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8540328Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8540331Z 2025-11-03T16:37:00.8540426Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8540620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8540680Z res = mod(**inputs) 2025-11-03T16:37:00.8540941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8541015Z outputs = self.mobilebert( 2025-11-03T16:37:00.8541279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8541351Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8541614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8541687Z layer_outputs = layer_module( 2025-11-03T16:37:00.8541951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8542053Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8542328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8542431Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8542717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8542824Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8542829Z 2025-11-03T16:37:00.8542925Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8543118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8543177Z res = mod(**inputs) 2025-11-03T16:37:00.8543450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8543517Z outputs = self.mobilebert( 2025-11-03T16:37:00.8543794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8543861Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8544133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8544206Z layer_outputs = layer_module( 2025-11-03T16:37:00.8544476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8544586Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8544852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8544969Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8545242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8545322Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8545325Z 2025-11-03T16:37:00.8545429Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8545618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8545685Z res = mod(**inputs) 2025-11-03T16:37:00.8545950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8546032Z outputs = self.mobilebert( 2025-11-03T16:37:00.8546303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8546371Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8546642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8546708Z layer_outputs = layer_module( 2025-11-03T16:37:00.8546971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8547067Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8547335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8547463Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8547730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8547851Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8548132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8548219Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8548222Z 2025-11-03T16:37:00.8548326Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8548528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8548596Z res = mod(**inputs) 2025-11-03T16:37:00.8548857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8548926Z outputs = self.mobilebert( 2025-11-03T16:37:00.8549197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8549265Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8549534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8549600Z layer_outputs = layer_module( 2025-11-03T16:37:00.8549869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8549983Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8550249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8550351Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8550354Z 2025-11-03T16:37:00.8550451Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8550642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8550701Z res = mod(**inputs) 2025-11-03T16:37:00.8550972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8551039Z outputs = self.mobilebert( 2025-11-03T16:37:00.8551302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8551378Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8551642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8551733Z layer_outputs = layer_module( 2025-11-03T16:37:00.8551997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8552110Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8552384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8552491Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8552495Z 2025-11-03T16:37:00.8552599Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8552786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8552855Z res = mod(**inputs) 2025-11-03T16:37:00.8553121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8553189Z outputs = self.mobilebert( 2025-11-03T16:37:00.8553460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8553528Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8553813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8553881Z layer_outputs = layer_module( 2025-11-03T16:37:00.8554219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8554389Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8554685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8554785Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8554791Z 2025-11-03T16:37:00.8554891Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8555089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8555153Z res = mod(**inputs) 2025-11-03T16:37:00.8555426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8555504Z outputs = self.mobilebert( 2025-11-03T16:37:00.8555786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8555864Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8556130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8556197Z layer_outputs = layer_module( 2025-11-03T16:37:00.8556471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8556647Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8556931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8557052Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8557337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8557426Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8557429Z 2025-11-03T16:37:00.8557528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8557731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8557811Z res = mod(**inputs) 2025-11-03T16:37:00.8558093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8558160Z outputs = self.mobilebert( 2025-11-03T16:37:00.8558433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8558510Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8558787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8558861Z layer_outputs = layer_module( 2025-11-03T16:37:00.8559127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8559280Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8559548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8559670Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8559942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8560034Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8560038Z 2025-11-03T16:37:00.8560141Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8560324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8560392Z res = mod(**inputs) 2025-11-03T16:37:00.8560670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8560737Z outputs = self.mobilebert( 2025-11-03T16:37:00.8561006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8561074Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8561340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8561406Z layer_outputs = layer_module( 2025-11-03T16:37:00.8561668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8561822Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8562083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8562205Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8562469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8562606Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8562873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8562962Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8562965Z 2025-11-03T16:37:00.8563069Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8563257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8563323Z res = mod(**inputs) 2025-11-03T16:37:00.8563588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8563654Z outputs = self.mobilebert( 2025-11-03T16:37:00.8563927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8564040Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8564315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8564383Z layer_outputs = layer_module( 2025-11-03T16:37:00.8564658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8564813Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8565080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8565193Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8565459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8565546Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8565549Z 2025-11-03T16:37:00.8565643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8565853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8565922Z res = mod(**inputs) 2025-11-03T16:37:00.8566187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8566261Z outputs = self.mobilebert( 2025-11-03T16:37:00.8566541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8566619Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8566882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8566951Z layer_outputs = layer_module( 2025-11-03T16:37:00.8567224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8567305Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8567579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8567646Z self_outputs = self.self( 2025-11-03T16:37:00.8567912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8567987Z self.value(value_tensor) 2025-11-03T16:37:00.8567990Z 2025-11-03T16:37:00.8568088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8568282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8568360Z res = mod(**inputs) 2025-11-03T16:37:00.8568638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8568704Z outputs = self.mobilebert( 2025-11-03T16:37:00.8568977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8569054Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8569323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8569397Z layer_outputs = layer_module( 2025-11-03T16:37:00.8569670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8569821Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8570111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8570216Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8570499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8570577Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8570581Z 2025-11-03T16:37:00.8570684Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8570873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8570932Z res = mod(**inputs) 2025-11-03T16:37:00.8571208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8571275Z outputs = self.mobilebert( 2025-11-03T16:37:00.8571549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8571618Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8571900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8571976Z layer_outputs = layer_module( 2025-11-03T16:37:00.8572241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8572397Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8572683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8572796Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8573072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8573153Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8573433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8573519Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8573523Z 2025-11-03T16:37:00.8573624Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8573813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8573881Z res = mod(**inputs) 2025-11-03T16:37:00.8574150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8574216Z outputs = self.mobilebert( 2025-11-03T16:37:00.8574494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8574575Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8574851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8574919Z layer_outputs = layer_module( 2025-11-03T16:37:00.8575179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8575266Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8575529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8575603Z self_outputs = self.self( 2025-11-03T16:37:00.8575864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8575960Z self.query(query_tensor) 2025-11-03T16:37:00.8575964Z 2025-11-03T16:37:00.8576058Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8576245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8576317Z res = mod(**inputs) 2025-11-03T16:37:00.8576588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8576661Z outputs = self.mobilebert( 2025-11-03T16:37:00.8576929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8576997Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8577275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8577344Z layer_outputs = layer_module( 2025-11-03T16:37:00.8577620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8577701Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8577980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8578055Z self_outputs = self.self( 2025-11-03T16:37:00.8578327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8578400Z self.key(key_tensor) 2025-11-03T16:37:00.8578404Z 2025-11-03T16:37:00.8578506Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8578591Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8578690Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8578884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8578956Z res = mod(**inputs) 2025-11-03T16:37:00.8579227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8579303Z outputs = self.mobilebert( 2025-11-03T16:37:00.8579572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8579640Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8579917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8579986Z layer_outputs = layer_module( 2025-11-03T16:37:00.8580263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8580360Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8580638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8580757Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8581027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8581115Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8581119Z 2025-11-03T16:37:00.8581217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8581415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8581479Z res = mod(**inputs) 2025-11-03T16:37:00.8581746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8581841Z outputs = self.mobilebert( 2025-11-03T16:37:00.8582111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8582188Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8582475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8582553Z layer_outputs = layer_module( 2025-11-03T16:37:00.8582827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8582909Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8583185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8583302Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8583581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8583702Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8583991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8584088Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8584092Z 2025-11-03T16:37:00.8584191Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8584386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8584447Z res = mod(**inputs) 2025-11-03T16:37:00.8584738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8584809Z outputs = self.mobilebert( 2025-11-03T16:37:00.8585082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8585159Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8585435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8585511Z layer_outputs = layer_module( 2025-11-03T16:37:00.8585784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8585878Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8586159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8586267Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8586549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8586649Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8586653Z 2025-11-03T16:37:00.8586757Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8586952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8587014Z res = mod(**inputs) 2025-11-03T16:37:00.8587295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8587364Z outputs = self.mobilebert( 2025-11-03T16:37:00.8587645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8587714Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8587988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8588080Z layer_outputs = layer_module( 2025-11-03T16:37:00.8588350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8588449Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8588721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8588837Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8589107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8589218Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8589222Z 2025-11-03T16:37:00.8589330Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8589523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8589592Z res = mod(**inputs) 2025-11-03T16:37:00.8589863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8589948Z outputs = self.mobilebert( 2025-11-03T16:37:00.8590231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8590300Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8590579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8590662Z layer_outputs = layer_module( 2025-11-03T16:37:00.8590942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8591036Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8591309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8591436Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8591710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8591797Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8591801Z 2025-11-03T16:37:00.8591898Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8592091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8592161Z res = mod(**inputs) 2025-11-03T16:37:00.8592436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8592529Z outputs = self.mobilebert( 2025-11-03T16:37:00.8592806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8592881Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8593154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8593221Z layer_outputs = layer_module( 2025-11-03T16:37:00.8593505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8593596Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8593876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8593996Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8594365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8594491Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8594763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8594861Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8594864Z 2025-11-03T16:37:00.8594963Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8595159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8595222Z res = mod(**inputs) 2025-11-03T16:37:00.8595500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8595580Z outputs = self.mobilebert( 2025-11-03T16:37:00.8595851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8595929Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8596217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8596287Z layer_outputs = layer_module( 2025-11-03T16:37:00.8596571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8596661Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8596959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8597077Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8597357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8597435Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8597439Z 2025-11-03T16:37:00.8597535Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8597730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8597790Z res = mod(**inputs) 2025-11-03T16:37:00.8598060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8598127Z outputs = self.mobilebert( 2025-11-03T16:37:00.8598397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8598465Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8598734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8599016Z layer_outputs = layer_module( 2025-11-03T16:37:00.8599291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8599389Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8599652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8599753Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8600026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8600132Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8600135Z 2025-11-03T16:37:00.8600241Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8600446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8600515Z res = mod(**inputs) 2025-11-03T16:37:00.8600785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8600856Z outputs = self.mobilebert( 2025-11-03T16:37:00.8601130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8601196Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8601470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8601538Z layer_outputs = layer_module( 2025-11-03T16:37:00.8601803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8601903Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8602169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8602294Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8602586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8602674Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8602677Z 2025-11-03T16:37:00.8602775Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8602979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8603051Z res = mod(**inputs) 2025-11-03T16:37:00.8603313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8603390Z outputs = self.mobilebert( 2025-11-03T16:37:00.8603652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8603719Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8603991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8604057Z layer_outputs = layer_module( 2025-11-03T16:37:00.8604330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8604417Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8604689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8604805Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8605083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8605205Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8605475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8605567Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8605570Z 2025-11-03T16:37:00.8605666Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8605853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8605922Z res = mod(**inputs) 2025-11-03T16:37:00.8606184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8606277Z outputs = self.mobilebert( 2025-11-03T16:37:00.8606545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8606621Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8606891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8606955Z layer_outputs = layer_module( 2025-11-03T16:37:00.8607228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8607315Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8607596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8607702Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8607972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8608055Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8608059Z 2025-11-03T16:37:00.8608152Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8608367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8608429Z res = mod(**inputs) 2025-11-03T16:37:00.8608703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8608768Z outputs = self.mobilebert( 2025-11-03T16:37:00.8609055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8609134Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8609406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8609483Z layer_outputs = layer_module( 2025-11-03T16:37:00.8609753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8609844Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8610122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8610229Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8610507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8610616Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8610621Z 2025-11-03T16:37:00.8610738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8610924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8610985Z res = mod(**inputs) 2025-11-03T16:37:00.8611258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8611326Z outputs = self.mobilebert( 2025-11-03T16:37:00.8611600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8611667Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8611934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8612007Z layer_outputs = layer_module( 2025-11-03T16:37:00.8612274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8612388Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8612655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8612780Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8613047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8613125Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8613128Z 2025-11-03T16:37:00.8613338Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8613542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8613612Z res = mod(**inputs) 2025-11-03T16:37:00.8613882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8613961Z outputs = self.mobilebert( 2025-11-03T16:37:00.8614231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8614301Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8614613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8614681Z layer_outputs = layer_module( 2025-11-03T16:37:00.8614952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8615062Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8615325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8615452Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8615718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8615839Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8616103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8616212Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8616216Z 2025-11-03T16:37:00.8616312Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8616502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8616572Z res = mod(**inputs) 2025-11-03T16:37:00.8616835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8616935Z outputs = self.mobilebert( 2025-11-03T16:37:00.8617197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8617263Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8617532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8617598Z layer_outputs = layer_module( 2025-11-03T16:37:00.8617867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8617979Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8618253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8618354Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8618359Z 2025-11-03T16:37:00.8618456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8618646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8618705Z res = mod(**inputs) 2025-11-03T16:37:00.8618977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8619043Z outputs = self.mobilebert( 2025-11-03T16:37:00.8619307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8619384Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8619647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8619721Z layer_outputs = layer_module( 2025-11-03T16:37:00.8619989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8620111Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8620391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8620498Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8620502Z 2025-11-03T16:37:00.8620606Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8620792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8620860Z res = mod(**inputs) 2025-11-03T16:37:00.8621142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8621212Z outputs = self.mobilebert( 2025-11-03T16:37:00.8621487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8621555Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8621825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8621890Z layer_outputs = layer_module( 2025-11-03T16:37:00.8622158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8622307Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8622571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8622666Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8622685Z 2025-11-03T16:37:00.8622781Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8622972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8623033Z res = mod(**inputs) 2025-11-03T16:37:00.8623301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8623375Z outputs = self.mobilebert( 2025-11-03T16:37:00.8623638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8623713Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8623981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8624053Z layer_outputs = layer_module( 2025-11-03T16:37:00.8624316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8624491Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8624764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8624882Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8625154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8625240Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8625243Z 2025-11-03T16:37:00.8625346Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8625539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8625597Z res = mod(**inputs) 2025-11-03T16:37:00.8625871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8625937Z outputs = self.mobilebert( 2025-11-03T16:37:00.8626207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8626289Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8626550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8626626Z layer_outputs = layer_module( 2025-11-03T16:37:00.8626904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8627060Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8627326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8627453Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8627721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8627803Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8627806Z 2025-11-03T16:37:00.8627912Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8628100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8628168Z res = mod(**inputs) 2025-11-03T16:37:00.8628437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8628505Z outputs = self.mobilebert( 2025-11-03T16:37:00.8628780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8628866Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8629143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8629211Z layer_outputs = layer_module( 2025-11-03T16:37:00.8629484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8629633Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8629904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8630027Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8630294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8630432Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8630702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8630791Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8630801Z 2025-11-03T16:37:00.8630897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8631086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8631155Z res = mod(**inputs) 2025-11-03T16:37:00.8631424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8631507Z outputs = self.mobilebert( 2025-11-03T16:37:00.8631776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8631846Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8632118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8632200Z layer_outputs = layer_module( 2025-11-03T16:37:00.8632475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8632628Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8632917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8633023Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8633288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8633378Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8633381Z 2025-11-03T16:37:00.8633477Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8633671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8633734Z res = mod(**inputs) 2025-11-03T16:37:00.8634055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8634147Z outputs = self.mobilebert( 2025-11-03T16:37:00.8634418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8634497Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8634769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8634865Z layer_outputs = layer_module( 2025-11-03T16:37:00.8635137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8635222Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8635502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8635572Z self_outputs = self.self( 2025-11-03T16:37:00.8635851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8635922Z self.value(value_tensor) 2025-11-03T16:37:00.8635925Z 2025-11-03T16:37:00.8636025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8636236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8636317Z res = mod(**inputs) 2025-11-03T16:37:00.8636591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8636657Z outputs = self.mobilebert( 2025-11-03T16:37:00.8636935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8637013Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8637284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8637362Z layer_outputs = layer_module( 2025-11-03T16:37:00.8637641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8637804Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8638080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8638186Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8638479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8638561Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8638565Z 2025-11-03T16:37:00.8638671Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8638861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8638929Z res = mod(**inputs) 2025-11-03T16:37:00.8639214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8639284Z outputs = self.mobilebert( 2025-11-03T16:37:00.8639564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8639636Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8639913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8639982Z layer_outputs = layer_module( 2025-11-03T16:37:00.8640250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8640412Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8640689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8640803Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8641088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8641176Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8641466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8641554Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8641557Z 2025-11-03T16:37:00.8641662Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8641851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8641918Z res = mod(**inputs) 2025-11-03T16:37:00.8642190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8642258Z outputs = self.mobilebert( 2025-11-03T16:37:00.8642553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8642622Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8642903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8642972Z layer_outputs = layer_module( 2025-11-03T16:37:00.8643253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8643337Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8643611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8643689Z self_outputs = self.self( 2025-11-03T16:37:00.8643962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8644040Z self.query(query_tensor) 2025-11-03T16:37:00.8644043Z 2025-11-03T16:37:00.8644143Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8644335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8644405Z res = mod(**inputs) 2025-11-03T16:37:00.8644696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8644771Z outputs = self.mobilebert( 2025-11-03T16:37:00.8645041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8645152Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8645423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8645496Z layer_outputs = layer_module( 2025-11-03T16:37:00.8645771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8645854Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8646132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8646201Z self_outputs = self.self( 2025-11-03T16:37:00.8646468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8646541Z self.key(key_tensor) 2025-11-03T16:37:00.8646545Z 2025-11-03T16:37:00.8646633Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8646717Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8646814Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8646996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8647088Z res = mod(**inputs) 2025-11-03T16:37:00.8647355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8647426Z outputs = self.mobilebert( 2025-11-03T16:37:00.8647702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8647780Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8648047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8648113Z layer_outputs = layer_module( 2025-11-03T16:37:00.8648389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8648486Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8648758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8648875Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8649144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8649233Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8649236Z 2025-11-03T16:37:00.8649333Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8649527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8649591Z res = mod(**inputs) 2025-11-03T16:37:00.8649863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8649931Z outputs = self.mobilebert( 2025-11-03T16:37:00.8650195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8650272Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8697232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8697434Z layer_outputs = layer_module( 2025-11-03T16:37:00.8697770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8697872Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8698189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8698324Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8698614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8698751Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8699027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8699125Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8699132Z 2025-11-03T16:37:00.8699250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8699514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8699602Z res = mod(**inputs) 2025-11-03T16:37:00.8699945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8700052Z outputs = self.mobilebert( 2025-11-03T16:37:00.8700368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8700447Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8700727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8700796Z layer_outputs = layer_module( 2025-11-03T16:37:00.8701066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8701158Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8701424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8701545Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8701810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8701974Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8701980Z 2025-11-03T16:37:00.8702087Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8702293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8702358Z res = mod(**inputs) 2025-11-03T16:37:00.8702646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8702729Z outputs = self.mobilebert( 2025-11-03T16:37:00.8703022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8703104Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8703397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8703471Z layer_outputs = layer_module( 2025-11-03T16:37:00.8703771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8703864Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8704197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8704310Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8704587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8704713Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8704718Z 2025-11-03T16:37:00.8704818Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8705023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8705083Z res = mod(**inputs) 2025-11-03T16:37:00.8705359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8705429Z outputs = self.mobilebert( 2025-11-03T16:37:00.8705698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8705777Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8706040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8706113Z layer_outputs = layer_module( 2025-11-03T16:37:00.8706375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8706487Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8706753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8706875Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8707153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8707236Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8707239Z 2025-11-03T16:37:00.8707344Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8707533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8707596Z res = mod(**inputs) 2025-11-03T16:37:00.8707867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8707954Z outputs = self.mobilebert( 2025-11-03T16:37:00.8708226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8708295Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8708570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8708636Z layer_outputs = layer_module( 2025-11-03T16:37:00.8708901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8708998Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8709267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8709394Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8709660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8709774Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8710063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8710156Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8710160Z 2025-11-03T16:37:00.8710266Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8710457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8710526Z res = mod(**inputs) 2025-11-03T16:37:00.8710810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8710880Z outputs = self.mobilebert( 2025-11-03T16:37:00.8711151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8711221Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8711494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8711560Z layer_outputs = layer_module( 2025-11-03T16:37:00.8711823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8711918Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8712185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8712300Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8712565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8712667Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8712671Z 2025-11-03T16:37:00.8712766Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8712954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8713021Z res = mod(**inputs) 2025-11-03T16:37:00.8713474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8713557Z outputs = self.mobilebert( 2025-11-03T16:37:00.8713841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8713913Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8714258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8714394Z layer_outputs = layer_module( 2025-11-03T16:37:00.8714688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8714784Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8715079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8715191Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8715481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8715597Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8715601Z 2025-11-03T16:37:00.8715699Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8715898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8715960Z res = mod(**inputs) 2025-11-03T16:37:00.8716235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8716328Z outputs = self.mobilebert( 2025-11-03T16:37:00.8716597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8716677Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8716956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8717056Z layer_outputs = layer_module( 2025-11-03T16:37:00.8717322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8717415Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8717689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8717808Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8718084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8718165Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8718168Z 2025-11-03T16:37:00.8718273Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8718462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8718524Z res = mod(**inputs) 2025-11-03T16:37:00.8718795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8718887Z outputs = self.mobilebert( 2025-11-03T16:37:00.8719168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8719237Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8719514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8719589Z layer_outputs = layer_module( 2025-11-03T16:37:00.8719876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8719971Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8720247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8720392Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8720661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8720777Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8721059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8721145Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8721148Z 2025-11-03T16:37:00.8721248Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8721431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8721491Z res = mod(**inputs) 2025-11-03T16:37:00.8721755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8721822Z outputs = self.mobilebert( 2025-11-03T16:37:00.8722083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8722150Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8722449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8722515Z layer_outputs = layer_module( 2025-11-03T16:37:00.8722776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8722870Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8723140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8723251Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8723514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8723593Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8723602Z 2025-11-03T16:37:00.8723696Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8723880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8723945Z res = mod(**inputs) 2025-11-03T16:37:00.8724204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8724276Z outputs = self.mobilebert( 2025-11-03T16:37:00.8724547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8724615Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8724909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8724976Z layer_outputs = layer_module( 2025-11-03T16:37:00.8725244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8725333Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8725592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8725703Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8725960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8726070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8726074Z 2025-11-03T16:37:00.8726182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8726366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8726425Z res = mod(**inputs) 2025-11-03T16:37:00.8726682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8726752Z outputs = self.mobilebert( 2025-11-03T16:37:00.8727022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8727087Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8727341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8727404Z layer_outputs = layer_module( 2025-11-03T16:37:00.8727664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8727749Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8728008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8728134Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8728393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8728469Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8728473Z 2025-11-03T16:37:00.8728563Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8728759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8728817Z res = mod(**inputs) 2025-11-03T16:37:00.8729081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8729146Z outputs = self.mobilebert( 2025-11-03T16:37:00.8729404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8729478Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8729736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8729808Z layer_outputs = layer_module( 2025-11-03T16:37:00.8730066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8730160Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8730417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8730547Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8730815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8730927Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8731194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8731279Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8731282Z 2025-11-03T16:37:00.8731385Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8731569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8731628Z res = mod(**inputs) 2025-11-03T16:37:00.8731891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8731977Z outputs = self.mobilebert( 2025-11-03T16:37:00.8732250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8732318Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8732589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8732659Z layer_outputs = layer_module( 2025-11-03T16:37:00.8732925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8733047Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8733318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8733407Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8733411Z 2025-11-03T16:37:00.8733506Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8733694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8733764Z res = mod(**inputs) 2025-11-03T16:37:00.8734046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8734121Z outputs = self.mobilebert( 2025-11-03T16:37:00.8734381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8734448Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8734728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8734795Z layer_outputs = layer_module( 2025-11-03T16:37:00.8735063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8735173Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8735442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8735545Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8735548Z 2025-11-03T16:37:00.8735644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8735834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8735892Z res = mod(**inputs) 2025-11-03T16:37:00.8736157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8736223Z outputs = self.mobilebert( 2025-11-03T16:37:00.8736492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8736566Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8736821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8736891Z layer_outputs = layer_module( 2025-11-03T16:37:00.8737145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8737301Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8737558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8737645Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8737661Z 2025-11-03T16:37:00.8737767Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8737944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8738008Z res = mod(**inputs) 2025-11-03T16:37:00.8738263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8738326Z outputs = self.mobilebert( 2025-11-03T16:37:00.8738586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8738648Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8738907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8738972Z layer_outputs = layer_module( 2025-11-03T16:37:00.8739237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8739385Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8739642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8739777Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8740040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8740131Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8740135Z 2025-11-03T16:37:00.8740230Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8740426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8740495Z res = mod(**inputs) 2025-11-03T16:37:00.8740756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8740830Z outputs = self.mobilebert( 2025-11-03T16:37:00.8741086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8741159Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8741418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8741483Z layer_outputs = layer_module( 2025-11-03T16:37:00.8741750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8741895Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8742160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8742292Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8742555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8742635Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8742639Z 2025-11-03T16:37:00.8742732Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8742919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8742973Z res = mod(**inputs) 2025-11-03T16:37:00.8743236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8743302Z outputs = self.mobilebert( 2025-11-03T16:37:00.8743556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8743649Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8743913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8743986Z layer_outputs = layer_module( 2025-11-03T16:37:00.8744256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8744410Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8744681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8744797Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8745081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8745195Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8745452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8745552Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8745555Z 2025-11-03T16:37:00.8745651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8745842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8745903Z res = mod(**inputs) 2025-11-03T16:37:00.8746201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8746271Z outputs = self.mobilebert( 2025-11-03T16:37:00.8746546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8746618Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8746890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8746965Z layer_outputs = layer_module( 2025-11-03T16:37:00.8747235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8747400Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8747667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8747773Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8748054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8748150Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8748154Z 2025-11-03T16:37:00.8748258Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8748448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8748518Z res = mod(**inputs) 2025-11-03T16:37:00.8748785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8748853Z outputs = self.mobilebert( 2025-11-03T16:37:00.8749124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8749194Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8749459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8749554Z layer_outputs = layer_module( 2025-11-03T16:37:00.8749825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8749916Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8750183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8750259Z self_outputs = self.self( 2025-11-03T16:37:00.8750525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8750598Z self.value(value_tensor) 2025-11-03T16:37:00.8750601Z 2025-11-03T16:37:00.8750716Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8750903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8750974Z res = mod(**inputs) 2025-11-03T16:37:00.8751238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8751303Z outputs = self.mobilebert( 2025-11-03T16:37:00.8751587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8751656Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8751926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8751993Z layer_outputs = layer_module( 2025-11-03T16:37:00.8752274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8752432Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8752702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8752812Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8753074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8753154Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8753157Z 2025-11-03T16:37:00.8753249Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8753435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8753494Z res = mod(**inputs) 2025-11-03T16:37:00.8753758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8753822Z outputs = self.mobilebert( 2025-11-03T16:37:00.8754187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8754264Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8754542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8754607Z layer_outputs = layer_module( 2025-11-03T16:37:00.8754886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8755041Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8755388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8755493Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8755782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8755868Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8756129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8756220Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8756223Z 2025-11-03T16:37:00.8756319Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8756509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8756566Z res = mod(**inputs) 2025-11-03T16:37:00.8756826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8756893Z outputs = self.mobilebert( 2025-11-03T16:37:00.8757155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8757222Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8757487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8757568Z layer_outputs = layer_module( 2025-11-03T16:37:00.8757840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8757922Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8758199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8758279Z self_outputs = self.self( 2025-11-03T16:37:00.8758550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8758620Z self.query(query_tensor) 2025-11-03T16:37:00.8758623Z 2025-11-03T16:37:00.8758721Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8758919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8758979Z res = mod(**inputs) 2025-11-03T16:37:00.8759254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8759319Z outputs = self.mobilebert( 2025-11-03T16:37:00.8759585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8759661Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8759927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8760016Z layer_outputs = layer_module( 2025-11-03T16:37:00.8760287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8760377Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8760647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8760713Z self_outputs = self.self( 2025-11-03T16:37:00.8760987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8761051Z self.key(key_tensor) 2025-11-03T16:37:00.8761055Z 2025-11-03T16:37:00.8761144Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8761222Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8761321Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8761535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8761597Z res = mod(**inputs) 2025-11-03T16:37:00.8761873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8761937Z outputs = self.mobilebert( 2025-11-03T16:37:00.8762203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8762280Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8762544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8762619Z layer_outputs = layer_module( 2025-11-03T16:37:00.8762886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8762975Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8763243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8763361Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8763652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8763732Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8763735Z 2025-11-03T16:37:00.8763838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8764024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8764084Z res = mod(**inputs) 2025-11-03T16:37:00.8764373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8764443Z outputs = self.mobilebert( 2025-11-03T16:37:00.8764740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8764811Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8765093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8765162Z layer_outputs = layer_module( 2025-11-03T16:37:00.8765432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8765519Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8765791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8765917Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8766212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8766334Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8766618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8766706Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8766709Z 2025-11-03T16:37:00.8766814Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8767009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8767076Z res = mod(**inputs) 2025-11-03T16:37:00.8767363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8767429Z outputs = self.mobilebert( 2025-11-03T16:37:00.8767723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8767792Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8768081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8768149Z layer_outputs = layer_module( 2025-11-03T16:37:00.8768423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8768524Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8768799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8768915Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8769190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8769282Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8769285Z 2025-11-03T16:37:00.8769383Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8769597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8769668Z res = mod(**inputs) 2025-11-03T16:37:00.8769941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8770012Z outputs = self.mobilebert( 2025-11-03T16:37:00.8770303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8770372Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8770649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8770720Z layer_outputs = layer_module( 2025-11-03T16:37:00.8771001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8771092Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8771374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8771483Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8771754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8771873Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8771876Z 2025-11-03T16:37:00.8771974Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8772190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8772251Z res = mod(**inputs) 2025-11-03T16:37:00.8772527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8772595Z outputs = self.mobilebert( 2025-11-03T16:37:00.8772864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8772941Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8773209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8773285Z layer_outputs = layer_module( 2025-11-03T16:37:00.8773556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8773669Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8773956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8774078Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8774365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8774448Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8774451Z 2025-11-03T16:37:00.8774554Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8774746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8774810Z res = mod(**inputs) 2025-11-03T16:37:00.8775090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8775161Z outputs = self.mobilebert( 2025-11-03T16:37:00.8775440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8775508Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8775800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8775879Z layer_outputs = layer_module( 2025-11-03T16:37:00.8776152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8776252Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8776546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8776676Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8776951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8777070Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8777347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8777434Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8777438Z 2025-11-03T16:37:00.8777544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8777734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8777796Z res = mod(**inputs) 2025-11-03T16:37:00.8778074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8778144Z outputs = self.mobilebert( 2025-11-03T16:37:00.8778435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8778503Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8778778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8778845Z layer_outputs = layer_module( 2025-11-03T16:37:00.8779112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8779201Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8779466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8779572Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8779853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8779934Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8779944Z 2025-11-03T16:37:00.8780043Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8780236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8780304Z res = mod(**inputs) 2025-11-03T16:37:00.8780574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8780651Z outputs = self.mobilebert( 2025-11-03T16:37:00.8780928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8780997Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8781275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8781345Z layer_outputs = layer_module( 2025-11-03T16:37:00.8781620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8781726Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8781998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8782114Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8782385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8782515Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8782519Z 2025-11-03T16:37:00.8782620Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8782841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8782905Z res = mod(**inputs) 2025-11-03T16:37:00.8783178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8783255Z outputs = self.mobilebert( 2025-11-03T16:37:00.8783527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8783604Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8783876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8783943Z layer_outputs = layer_module( 2025-11-03T16:37:00.8784220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8784327Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8784601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8784719Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8784995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8785077Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8785080Z 2025-11-03T16:37:00.8785178Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8785377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8785438Z res = mod(**inputs) 2025-11-03T16:37:00.8785717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8785803Z outputs = self.mobilebert( 2025-11-03T16:37:00.8786077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8786155Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8786425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8786501Z layer_outputs = layer_module( 2025-11-03T16:37:00.8786770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8786874Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8787131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8787244Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8787511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8787623Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8787900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8787985Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8787989Z 2025-11-03T16:37:00.8788087Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8788267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8788340Z res = mod(**inputs) 2025-11-03T16:37:00.8788605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8788671Z outputs = self.mobilebert( 2025-11-03T16:37:00.8788932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8788996Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8789254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8789325Z layer_outputs = layer_module( 2025-11-03T16:37:00.8789585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8789678Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8789937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8790044Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8790343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8790420Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8790423Z 2025-11-03T16:37:00.8790526Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8790709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8790775Z res = mod(**inputs) 2025-11-03T16:37:00.8791028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8791091Z outputs = self.mobilebert( 2025-11-03T16:37:00.8791352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8791418Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8791708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8791776Z layer_outputs = layer_module( 2025-11-03T16:37:00.8792041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8792127Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8792390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8792503Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8792766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8792879Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8792882Z 2025-11-03T16:37:00.8792978Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8793163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8793230Z res = mod(**inputs) 2025-11-03T16:37:00.8793512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8793588Z outputs = self.mobilebert( 2025-11-03T16:37:00.8793854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8793929Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8794283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8794357Z layer_outputs = layer_module( 2025-11-03T16:37:00.8794634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8794727Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8795008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8795137Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8795393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8795474Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8795477Z 2025-11-03T16:37:00.8795566Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8795753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8795812Z res = mod(**inputs) 2025-11-03T16:37:00.8796075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8796160Z outputs = self.mobilebert( 2025-11-03T16:37:00.8796422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8796499Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8796761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8796834Z layer_outputs = layer_module( 2025-11-03T16:37:00.8797095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8797182Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8797448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8797581Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8797847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8797957Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8798224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8798308Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8798311Z 2025-11-03T16:37:00.8798406Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8798596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8798657Z res = mod(**inputs) 2025-11-03T16:37:00.8798924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8798992Z outputs = self.mobilebert( 2025-11-03T16:37:00.8799252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8799326Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8799599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8799669Z layer_outputs = layer_module( 2025-11-03T16:37:00.8799920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8800030Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8800297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8800377Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8800382Z 2025-11-03T16:37:00.8800481Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8800660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8800725Z res = mod(**inputs) 2025-11-03T16:37:00.8800996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8801060Z outputs = self.mobilebert( 2025-11-03T16:37:00.8801322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8801388Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8801655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8801721Z layer_outputs = layer_module( 2025-11-03T16:37:00.8802002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8802111Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8802369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8802477Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8802480Z 2025-11-03T16:37:00.8802573Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8802756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8802812Z res = mod(**inputs) 2025-11-03T16:37:00.8803069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8803139Z outputs = self.mobilebert( 2025-11-03T16:37:00.8803411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8803484Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8803742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8803814Z layer_outputs = layer_module( 2025-11-03T16:37:00.8804072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8804216Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8804482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8804570Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8804574Z 2025-11-03T16:37:00.8804677Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8804858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8804922Z res = mod(**inputs) 2025-11-03T16:37:00.8805194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8805260Z outputs = self.mobilebert( 2025-11-03T16:37:00.8805525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8805593Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8805869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8805936Z layer_outputs = layer_module( 2025-11-03T16:37:00.8806193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8806348Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8806604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8806726Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8806982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8807067Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8807070Z 2025-11-03T16:37:00.8807163Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8807347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8807416Z res = mod(**inputs) 2025-11-03T16:37:00.8807695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8807766Z outputs = self.mobilebert( 2025-11-03T16:37:00.8808021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8808087Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8808350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8808414Z layer_outputs = layer_module( 2025-11-03T16:37:00.8808676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8808820Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8809087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8809219Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8809476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8809564Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8809567Z 2025-11-03T16:37:00.8809660Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8809845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8809900Z res = mod(**inputs) 2025-11-03T16:37:00.8810158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8810229Z outputs = self.mobilebert( 2025-11-03T16:37:00.8810485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8810560Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8810816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8810902Z layer_outputs = layer_module( 2025-11-03T16:37:00.8811163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8811304Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8811569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8811696Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8811966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8812079Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8812345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8812430Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8812433Z 2025-11-03T16:37:00.8812526Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8812715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8812772Z res = mod(**inputs) 2025-11-03T16:37:00.8813037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8813102Z outputs = self.mobilebert( 2025-11-03T16:37:00.8813514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8813636Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8813897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8813970Z layer_outputs = layer_module( 2025-11-03T16:37:00.8814232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8814387Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8814647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8814751Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8815019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8815121Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8815125Z 2025-11-03T16:37:00.8815229Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8815412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8815482Z res = mod(**inputs) 2025-11-03T16:37:00.8815742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8815806Z outputs = self.mobilebert( 2025-11-03T16:37:00.8816073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8816141Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8816407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8816475Z layer_outputs = layer_module( 2025-11-03T16:37:00.8816740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8816829Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8817108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8817179Z self_outputs = self.self( 2025-11-03T16:37:00.8817439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8817513Z self.value(value_tensor) 2025-11-03T16:37:00.8817516Z 2025-11-03T16:37:00.8817632Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8817814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8817883Z res = mod(**inputs) 2025-11-03T16:37:00.8818140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8818213Z outputs = self.mobilebert( 2025-11-03T16:37:00.8818469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8818535Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8818796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8818858Z layer_outputs = layer_module( 2025-11-03T16:37:00.8819130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8819276Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8819559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8819659Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8819931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8820006Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8820010Z 2025-11-03T16:37:00.8820103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8820291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8820350Z res = mod(**inputs) 2025-11-03T16:37:00.8820616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8820680Z outputs = self.mobilebert( 2025-11-03T16:37:00.8820964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8821032Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8821289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8821361Z layer_outputs = layer_module( 2025-11-03T16:37:00.8821619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8821769Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8822036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8822135Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8822399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8822479Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8822749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8822849Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8822852Z 2025-11-03T16:37:00.8822954Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8823135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8823193Z res = mod(**inputs) 2025-11-03T16:37:00.8823476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8823541Z outputs = self.mobilebert( 2025-11-03T16:37:00.8823810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8823879Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8824138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8824210Z layer_outputs = layer_module( 2025-11-03T16:37:00.8824474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8824558Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8824820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8824891Z self_outputs = self.self( 2025-11-03T16:37:00.8825147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8825230Z self.query(query_tensor) 2025-11-03T16:37:00.8825234Z 2025-11-03T16:37:00.8825333Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8825514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8825579Z res = mod(**inputs) 2025-11-03T16:37:00.8825848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8825911Z outputs = self.mobilebert( 2025-11-03T16:37:00.8826178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8826244Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8826507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8826589Z layer_outputs = layer_module( 2025-11-03T16:37:00.8826854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8826930Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8827186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8827255Z self_outputs = self.self( 2025-11-03T16:37:00.8827511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8827580Z self.key(key_tensor) 2025-11-03T16:37:00.8827583Z 2025-11-03T16:37:00.8827658Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8827732Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8827834Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8828019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8828085Z res = mod(**inputs) 2025-11-03T16:37:00.8828346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8828410Z outputs = self.mobilebert( 2025-11-03T16:37:00.8828689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8828755Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8829024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8829088Z layer_outputs = layer_module( 2025-11-03T16:37:00.8829368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8829446Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8829704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8829824Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8830087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8830173Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8830176Z 2025-11-03T16:37:00.8830271Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8830455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8830521Z res = mod(**inputs) 2025-11-03T16:37:00.8830795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8830886Z outputs = self.mobilebert( 2025-11-03T16:37:00.8831152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8831227Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8831497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8831563Z layer_outputs = layer_module( 2025-11-03T16:37:00.8831838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8831920Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8832195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8832311Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8832601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8832729Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8833001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8833095Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8833098Z 2025-11-03T16:37:00.8833193Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8833386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8833445Z res = mod(**inputs) 2025-11-03T16:37:00.8833759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8833835Z outputs = self.mobilebert( 2025-11-03T16:37:00.8834199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8834284Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8834561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8834650Z layer_outputs = layer_module( 2025-11-03T16:37:00.8834933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8835029Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8835352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8835460Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8835734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8835819Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8835823Z 2025-11-03T16:37:00.8835919Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8836118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8836179Z res = mod(**inputs) 2025-11-03T16:37:00.8836454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8836521Z outputs = self.mobilebert( 2025-11-03T16:37:00.8836791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8836866Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8837142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8837234Z layer_outputs = layer_module( 2025-11-03T16:37:00.8837502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8837599Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8837869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8837973Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8838248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8838359Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8838362Z 2025-11-03T16:37:00.8838464Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8838673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8838741Z res = mod(**inputs) 2025-11-03T16:37:00.8839008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8839073Z outputs = self.mobilebert( 2025-11-03T16:37:00.8839343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8839412Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8839682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8839747Z layer_outputs = layer_module( 2025-11-03T16:37:00.8840013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8840112Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8840379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8840503Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8840785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8840870Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8840874Z 2025-11-03T16:37:00.8840970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8841159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8841240Z res = mod(**inputs) 2025-11-03T16:37:00.8841509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8841584Z outputs = self.mobilebert( 2025-11-03T16:37:00.8841849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8841917Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8842193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8842259Z layer_outputs = layer_module( 2025-11-03T16:37:00.8842533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8842622Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8842896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8843013Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8843297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8843417Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8843687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8843779Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8843782Z 2025-11-03T16:37:00.8843877Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8844059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8844124Z res = mod(**inputs) 2025-11-03T16:37:00.8844377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8844466Z outputs = self.mobilebert( 2025-11-03T16:37:00.8844727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8844800Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8845061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8845128Z layer_outputs = layer_module( 2025-11-03T16:37:00.8845395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8845480Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8845748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8845850Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8846113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8846199Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8846203Z 2025-11-03T16:37:00.8846298Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8846501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8846563Z res = mod(**inputs) 2025-11-03T16:37:00.8846827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8846893Z outputs = self.mobilebert( 2025-11-03T16:37:00.8847165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8847241Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8847499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8847573Z layer_outputs = layer_module( 2025-11-03T16:37:00.8847829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8847918Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8848185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8848290Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8848554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8848657Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8848661Z 2025-11-03T16:37:00.8848765Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8848964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8849022Z res = mod(**inputs) 2025-11-03T16:37:00.8849287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8849354Z outputs = self.mobilebert( 2025-11-03T16:37:00.8849621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8849687Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8849945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8850016Z layer_outputs = layer_module( 2025-11-03T16:37:00.8850278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8850386Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8850640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8850761Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8851018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8851096Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8851100Z 2025-11-03T16:37:00.8851201Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8851384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8851448Z res = mod(**inputs) 2025-11-03T16:37:00.8851704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8851769Z outputs = self.mobilebert( 2025-11-03T16:37:00.8852032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8852098Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8852377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8852444Z layer_outputs = layer_module( 2025-11-03T16:37:00.8852708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8852794Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8853062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8853186Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8853443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8853562Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8853826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8853917Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8853921Z 2025-11-03T16:37:00.8854014Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8854198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8854264Z res = mod(**inputs) 2025-11-03T16:37:00.8854523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8854613Z outputs = self.mobilebert( 2025-11-03T16:37:00.8854869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8854934Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8855198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8855262Z layer_outputs = layer_module( 2025-11-03T16:37:00.8855524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8855607Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8855895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8855997Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8856268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8856351Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8856354Z 2025-11-03T16:37:00.8856446Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8856633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8856694Z res = mod(**inputs) 2025-11-03T16:37:00.8856949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8857021Z outputs = self.mobilebert( 2025-11-03T16:37:00.8857279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8857351Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8857609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8857682Z layer_outputs = layer_module( 2025-11-03T16:37:00.8857936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8858043Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8858307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8858408Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8858684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8858788Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8858792Z 2025-11-03T16:37:00.8858887Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8859078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8859136Z res = mod(**inputs) 2025-11-03T16:37:00.8859400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8859468Z outputs = self.mobilebert( 2025-11-03T16:37:00.8859737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8859802Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8860060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8860134Z layer_outputs = layer_module( 2025-11-03T16:37:00.8860392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8860505Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8860761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8860878Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8861142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8861219Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8861222Z 2025-11-03T16:37:00.8861323Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8861505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8861572Z res = mod(**inputs) 2025-11-03T16:37:00.8861826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8861908Z outputs = self.mobilebert( 2025-11-03T16:37:00.8862178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8862244Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8862509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8862575Z layer_outputs = layer_module( 2025-11-03T16:37:00.8862835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8862930Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8863196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8863320Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8863581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8863699Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8863977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8864063Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8864066Z 2025-11-03T16:37:00.8864167Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8864353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8864436Z res = mod(**inputs) 2025-11-03T16:37:00.8864704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8864772Z outputs = self.mobilebert( 2025-11-03T16:37:00.8865043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8865109Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8865381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8865448Z layer_outputs = layer_module( 2025-11-03T16:37:00.8865724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8865841Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8866115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8866204Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8866221Z 2025-11-03T16:37:00.8866320Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8866512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8866571Z res = mod(**inputs) 2025-11-03T16:37:00.8866837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8866909Z outputs = self.mobilebert( 2025-11-03T16:37:00.8867172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8867248Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8867517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8867589Z layer_outputs = layer_module( 2025-11-03T16:37:00.8867874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8867989Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8868269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8868374Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8868377Z 2025-11-03T16:37:00.8868478Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8868667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8868726Z res = mod(**inputs) 2025-11-03T16:37:00.8869002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8869067Z outputs = self.mobilebert( 2025-11-03T16:37:00.8869342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8869408Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8869697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8869763Z layer_outputs = layer_module( 2025-11-03T16:37:00.8870027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8870186Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8870469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8870568Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8870573Z 2025-11-03T16:37:00.8870670Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8870864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8870924Z res = mod(**inputs) 2025-11-03T16:37:00.8871189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8871262Z outputs = self.mobilebert( 2025-11-03T16:37:00.8871527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8871600Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8871868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8871933Z layer_outputs = layer_module( 2025-11-03T16:37:00.8872206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8872369Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8872644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8872762Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8873035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8873120Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8873124Z 2025-11-03T16:37:00.8873219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8873411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8873472Z res = mod(**inputs) 2025-11-03T16:37:00.8873767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8873836Z outputs = self.mobilebert( 2025-11-03T16:37:00.8874176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8874261Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8874535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8874614Z layer_outputs = layer_module( 2025-11-03T16:37:00.8874891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8875051Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8875324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8875450Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8875732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8875834Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8875837Z 2025-11-03T16:37:00.8875945Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8876138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8876201Z res = mod(**inputs) 2025-11-03T16:37:00.8876851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8876922Z outputs = self.mobilebert( 2025-11-03T16:37:00.8877202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8877273Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8877553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8877622Z layer_outputs = layer_module( 2025-11-03T16:37:00.8877901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8878059Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8878328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8878455Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8878732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8878876Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8879152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8879243Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8879247Z 2025-11-03T16:37:00.8879362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8879553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8879620Z res = mod(**inputs) 2025-11-03T16:37:00.8879898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8879965Z outputs = self.mobilebert( 2025-11-03T16:37:00.8880255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8880352Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8880632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8880699Z layer_outputs = layer_module( 2025-11-03T16:37:00.8880977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8881133Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8881408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8881525Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8881796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8881885Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8881889Z 2025-11-03T16:37:00.8881988Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8882178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8882264Z res = mod(**inputs) 2025-11-03T16:37:00.8882538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8882615Z outputs = self.mobilebert( 2025-11-03T16:37:00.8882894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8883016Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8883290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8883361Z layer_outputs = layer_module( 2025-11-03T16:37:00.8883642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8883725Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8884007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8884077Z self_outputs = self.self( 2025-11-03T16:37:00.8884349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8884426Z self.value(value_tensor) 2025-11-03T16:37:00.8884430Z 2025-11-03T16:37:00.8884528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8884728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8884808Z res = mod(**inputs) 2025-11-03T16:37:00.8885093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8885160Z outputs = self.mobilebert( 2025-11-03T16:37:00.8885433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8885510Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8885784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8885859Z layer_outputs = layer_module( 2025-11-03T16:37:00.8886133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8886288Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8886590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8886696Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8886976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8887053Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8887057Z 2025-11-03T16:37:00.8887161Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8887352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8887413Z res = mod(**inputs) 2025-11-03T16:37:00.8887693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8887761Z outputs = self.mobilebert( 2025-11-03T16:37:00.8888040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8888109Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8888384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8888475Z layer_outputs = layer_module( 2025-11-03T16:37:00.8888753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8888911Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8889204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8889316Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8889592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8889676Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8889959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8890048Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8890051Z 2025-11-03T16:37:00.8890158Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8890352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8890414Z res = mod(**inputs) 2025-11-03T16:37:00.8890697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8890766Z outputs = self.mobilebert( 2025-11-03T16:37:00.8891049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8891135Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8891414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8891484Z layer_outputs = layer_module( 2025-11-03T16:37:00.8891757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8891848Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8892123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8892212Z self_outputs = self.self( 2025-11-03T16:37:00.8892467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8892550Z self.query(query_tensor) 2025-11-03T16:37:00.8892560Z 2025-11-03T16:37:00.8892655Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8892835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8892901Z res = mod(**inputs) 2025-11-03T16:37:00.8893158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8893230Z outputs = self.mobilebert( 2025-11-03T16:37:00.8893484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8893549Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8893813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8893879Z layer_outputs = layer_module( 2025-11-03T16:37:00.8894144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8894221Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8894493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8894565Z self_outputs = self.self( 2025-11-03T16:37:00.8894824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8894893Z self.key(key_tensor) 2025-11-03T16:37:00.8894896Z 2025-11-03T16:37:00.8894973Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8895065Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8895162Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8895345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8895414Z res = mod(**inputs) 2025-11-03T16:37:00.8895674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8895758Z outputs = self.mobilebert( 2025-11-03T16:37:00.8896019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8896086Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8896353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8896417Z layer_outputs = layer_module( 2025-11-03T16:37:00.8896685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8896764Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8897034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8897155Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8897414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8897499Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8897502Z 2025-11-03T16:37:00.8897597Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8897785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8897844Z res = mod(**inputs) 2025-11-03T16:37:00.8898101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8898191Z outputs = self.mobilebert( 2025-11-03T16:37:00.8898451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8898524Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8898780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8898844Z layer_outputs = layer_module( 2025-11-03T16:37:00.8899109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8899186Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8899447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8899559Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8899826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8899943Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8900219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8900310Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8900313Z 2025-11-03T16:37:00.8900407Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8900594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8900652Z res = mod(**inputs) 2025-11-03T16:37:00.8900928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8900995Z outputs = self.mobilebert( 2025-11-03T16:37:00.8901251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8901325Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8901580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8901652Z layer_outputs = layer_module( 2025-11-03T16:37:00.8901913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8902001Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8902266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8902365Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8902629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8902736Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8902739Z 2025-11-03T16:37:00.8902839Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8903022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8903079Z res = mod(**inputs) 2025-11-03T16:37:00.8903343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8903407Z outputs = self.mobilebert( 2025-11-03T16:37:00.8903670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8903736Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8903993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8904080Z layer_outputs = layer_module( 2025-11-03T16:37:00.8904336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8904431Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8904691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8904800Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8905058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8905164Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8905167Z 2025-11-03T16:37:00.8905270Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8905454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8905521Z res = mod(**inputs) 2025-11-03T16:37:00.8905778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8905841Z outputs = self.mobilebert( 2025-11-03T16:37:00.8906123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8906191Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8906455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8906519Z layer_outputs = layer_module( 2025-11-03T16:37:00.8906796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8906885Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8907142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8907266Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8907524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8907608Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8907611Z 2025-11-03T16:37:00.8907705Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8907885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8907950Z res = mod(**inputs) 2025-11-03T16:37:00.8908207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8908297Z outputs = self.mobilebert( 2025-11-03T16:37:00.8908561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8908634Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8908898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8908965Z layer_outputs = layer_module( 2025-11-03T16:37:00.8909239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8909327Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8909613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8909732Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8910014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8910134Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8910400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8910494Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8910497Z 2025-11-03T16:37:00.8910593Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8910786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8910845Z res = mod(**inputs) 2025-11-03T16:37:00.8911110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8911185Z outputs = self.mobilebert( 2025-11-03T16:37:00.8911449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8911525Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8911809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8911878Z layer_outputs = layer_module( 2025-11-03T16:37:00.8912150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8912238Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8912530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8912635Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8912909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8912991Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8912994Z 2025-11-03T16:37:00.8913090Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8913432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8913497Z res = mod(**inputs) 2025-11-03T16:37:00.8913772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8913840Z outputs = self.mobilebert( 2025-11-03T16:37:00.8914157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8914247Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8914521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8914634Z layer_outputs = layer_module( 2025-11-03T16:37:00.8914903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8915006Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8915290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8915393Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8915663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8915770Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8915773Z 2025-11-03T16:37:00.8915876Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8916086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8916155Z res = mod(**inputs) 2025-11-03T16:37:00.8916430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8916498Z outputs = self.mobilebert( 2025-11-03T16:37:00.8916773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8916840Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8917113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8917182Z layer_outputs = layer_module( 2025-11-03T16:37:00.8917444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8917543Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8917807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8917930Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8918232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8918320Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8918323Z 2025-11-03T16:37:00.8918420Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8918605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8918694Z res = mod(**inputs) 2025-11-03T16:37:00.8918959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8919033Z outputs = self.mobilebert( 2025-11-03T16:37:00.8919298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8919366Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8919653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8919719Z layer_outputs = layer_module( 2025-11-03T16:37:00.8919987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8920075Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8920345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8920462Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8920742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8920867Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8921130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8921224Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8921228Z 2025-11-03T16:37:00.8921323Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8921506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8921574Z res = mod(**inputs) 2025-11-03T16:37:00.8921837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8921929Z outputs = self.mobilebert( 2025-11-03T16:37:00.8922195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8922270Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8922539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8922605Z layer_outputs = layer_module( 2025-11-03T16:37:00.8922877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8922965Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8923238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8923343Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8923610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8923696Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8923700Z 2025-11-03T16:37:00.8923797Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8924008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8924069Z res = mod(**inputs) 2025-11-03T16:37:00.8924341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8924407Z outputs = self.mobilebert( 2025-11-03T16:37:00.8924706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8924782Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8925039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8925112Z layer_outputs = layer_module( 2025-11-03T16:37:00.8925366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8925454Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8925718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8925818Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8926095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8926196Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8926199Z 2025-11-03T16:37:00.8926302Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8926499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8926557Z res = mod(**inputs) 2025-11-03T16:37:00.8926825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8926891Z outputs = self.mobilebert( 2025-11-03T16:37:00.8927157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8927224Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8927481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8927554Z layer_outputs = layer_module( 2025-11-03T16:37:00.8927811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8927923Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8928180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8928301Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8928562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8928641Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8928644Z 2025-11-03T16:37:00.8928746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8928929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8928998Z res = mod(**inputs) 2025-11-03T16:37:00.8929253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8929320Z outputs = self.mobilebert( 2025-11-03T16:37:00.8929585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8929652Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8929934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8930001Z layer_outputs = layer_module( 2025-11-03T16:37:00.8930267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8930352Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8930623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8930750Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8931005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8931124Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8931382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8931473Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8931476Z 2025-11-03T16:37:00.8931569Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8931750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8931817Z res = mod(**inputs) 2025-11-03T16:37:00.8932076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8932165Z outputs = self.mobilebert( 2025-11-03T16:37:00.8932423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8932488Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8932752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8932816Z layer_outputs = layer_module( 2025-11-03T16:37:00.8933079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8933189Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8933451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8933528Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8933549Z 2025-11-03T16:37:00.8933643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8933833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8933892Z res = mod(**inputs) 2025-11-03T16:37:00.8934157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8934222Z outputs = self.mobilebert( 2025-11-03T16:37:00.8934494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8934570Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8934838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8934911Z layer_outputs = layer_module( 2025-11-03T16:37:00.8935187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.8935305Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.8935589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8935718Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8935722Z 2025-11-03T16:37:00.8935828Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8936016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8936083Z res = mod(**inputs) 2025-11-03T16:37:00.8936374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8936441Z outputs = self.mobilebert( 2025-11-03T16:37:00.8936706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8936773Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8937036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8937101Z layer_outputs = layer_module( 2025-11-03T16:37:00.8937366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8937511Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8937769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.8937864Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.8937869Z 2025-11-03T16:37:00.8937981Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8938169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8938228Z res = mod(**inputs) 2025-11-03T16:37:00.8938490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8938562Z outputs = self.mobilebert( 2025-11-03T16:37:00.8938824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8938898Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8939161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8939233Z layer_outputs = layer_module( 2025-11-03T16:37:00.8939499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8939664Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8939932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.8940047Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.8940317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8940399Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8940402Z 2025-11-03T16:37:00.8940503Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8940690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8940748Z res = mod(**inputs) 2025-11-03T16:37:00.8941022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8941088Z outputs = self.mobilebert( 2025-11-03T16:37:00.8941361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8941441Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8941701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8941773Z layer_outputs = layer_module( 2025-11-03T16:37:00.8942033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8942199Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8942457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8942574Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8942841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.8942919Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8942922Z 2025-11-03T16:37:00.8943023Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8943204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8943269Z res = mod(**inputs) 2025-11-03T16:37:00.8943527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8943590Z outputs = self.mobilebert( 2025-11-03T16:37:00.8943858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8943938Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8944201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8944264Z layer_outputs = layer_module( 2025-11-03T16:37:00.8944524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.8944674Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.8944930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.8945050Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.8945306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.8945444Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8945704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8945787Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8945798Z 2025-11-03T16:37:00.8945892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8946075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8946140Z res = mod(**inputs) 2025-11-03T16:37:00.8946405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8946477Z outputs = self.mobilebert( 2025-11-03T16:37:00.8946736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8946804Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8947069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8947132Z layer_outputs = layer_module( 2025-11-03T16:37:00.8947422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8947573Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8947835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8947961Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8948219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8948305Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8948309Z 2025-11-03T16:37:00.8948403Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8948591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8948649Z res = mod(**inputs) 2025-11-03T16:37:00.8948907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8948980Z outputs = self.mobilebert( 2025-11-03T16:37:00.8949238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8949312Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8949567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8949651Z layer_outputs = layer_module( 2025-11-03T16:37:00.8949918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8949998Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8950263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8950330Z self_outputs = self.self( 2025-11-03T16:37:00.8950596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.8950663Z self.value(value_tensor) 2025-11-03T16:37:00.8950666Z 2025-11-03T16:37:00.8950763Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8950953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8951029Z res = mod(**inputs) 2025-11-03T16:37:00.8951292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8951357Z outputs = self.mobilebert( 2025-11-03T16:37:00.8951614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8951689Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8951946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8952017Z layer_outputs = layer_module( 2025-11-03T16:37:00.8952279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8952434Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8952695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.8952797Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.8953081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.8953158Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.8953161Z 2025-11-03T16:37:00.8953264Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8953446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8953504Z res = mod(**inputs) 2025-11-03T16:37:00.8953804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8953875Z outputs = self.mobilebert( 2025-11-03T16:37:00.8954223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8954299Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8954583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8954657Z layer_outputs = layer_module( 2025-11-03T16:37:00.8954933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.8955101Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.8955386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.8955499Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.8955829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.8955943Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.8956213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8956305Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8956308Z 2025-11-03T16:37:00.8956416Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8956611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8956681Z res = mod(**inputs) 2025-11-03T16:37:00.8956954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8957024Z outputs = self.mobilebert( 2025-11-03T16:37:00.8957307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8957397Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8957680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8957749Z layer_outputs = layer_module( 2025-11-03T16:37:00.8958030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8958112Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8958388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8958465Z self_outputs = self.self( 2025-11-03T16:37:00.8958740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.8958818Z self.query(query_tensor) 2025-11-03T16:37:00.8958821Z 2025-11-03T16:37:00.8958920Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8959113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8959182Z res = mod(**inputs) 2025-11-03T16:37:00.8959499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8959575Z outputs = self.mobilebert( 2025-11-03T16:37:00.8959849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8959920Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8960217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8960289Z layer_outputs = layer_module( 2025-11-03T16:37:00.8960568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8960647Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8960927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.8960993Z self_outputs = self.self( 2025-11-03T16:37:00.8961265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.8961336Z self.key(key_tensor) 2025-11-03T16:37:00.8961340Z 2025-11-03T16:37:00.8961417Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8961503Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.8961601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8961792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8961879Z res = mod(**inputs) 2025-11-03T16:37:00.8962149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8962224Z outputs = self.mobilebert( 2025-11-03T16:37:00.8962495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8962572Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8962844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8962910Z layer_outputs = layer_module( 2025-11-03T16:37:00.8963193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8963289Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8963571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8963688Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8963963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.8964050Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8964054Z 2025-11-03T16:37:00.8964151Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8964350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8964410Z res = mod(**inputs) 2025-11-03T16:37:00.8964691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8964760Z outputs = self.mobilebert( 2025-11-03T16:37:00.8965035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8965112Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8965400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8965476Z layer_outputs = layer_module( 2025-11-03T16:37:00.8965749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.8965830Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.8966133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.8966253Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.8966536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.8966659Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8966939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8967028Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8967032Z 2025-11-03T16:37:00.8967130Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8967329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8967389Z res = mod(**inputs) 2025-11-03T16:37:00.8967668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8967737Z outputs = self.mobilebert( 2025-11-03T16:37:00.8968022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8968115Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8968390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8968467Z layer_outputs = layer_module( 2025-11-03T16:37:00.8968744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8968844Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8969120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8969241Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8969516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8969620Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8969624Z 2025-11-03T16:37:00.8969726Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8969913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8969975Z res = mod(**inputs) 2025-11-03T16:37:00.8970247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8970312Z outputs = self.mobilebert( 2025-11-03T16:37:00.8970585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8970652Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8970923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8970991Z layer_outputs = layer_module( 2025-11-03T16:37:00.8971255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8971352Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8971632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8971746Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8972014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8972139Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8972150Z 2025-11-03T16:37:00.8972246Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8972433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8972502Z res = mod(**inputs) 2025-11-03T16:37:00.8972767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8972842Z outputs = self.mobilebert( 2025-11-03T16:37:00.8973111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8973177Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8973454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8973519Z layer_outputs = layer_module( 2025-11-03T16:37:00.8973796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8973888Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8974173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8974300Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8974564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8974652Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8974656Z 2025-11-03T16:37:00.8974750Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8974945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8975005Z res = mod(**inputs) 2025-11-03T16:37:00.8975274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8975365Z outputs = self.mobilebert( 2025-11-03T16:37:00.8975637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8975711Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8975979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8976046Z layer_outputs = layer_module( 2025-11-03T16:37:00.8976319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8976408Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8976683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8976801Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8977080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8977196Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8977507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8977599Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8977603Z 2025-11-03T16:37:00.8977699Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8977890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8977949Z res = mod(**inputs) 2025-11-03T16:37:00.8978235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8978303Z outputs = self.mobilebert( 2025-11-03T16:37:00.8978568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8978642Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8978907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8978980Z layer_outputs = layer_module( 2025-11-03T16:37:00.8979243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8979330Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8979601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8979705Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8979975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8980070Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8980073Z 2025-11-03T16:37:00.8980177Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8980364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8980422Z res = mod(**inputs) 2025-11-03T16:37:00.8980693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8980758Z outputs = self.mobilebert( 2025-11-03T16:37:00.8981027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8981096Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8981357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8981448Z layer_outputs = layer_module( 2025-11-03T16:37:00.8981710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8981806Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8982068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8982179Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8982441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8982548Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8982551Z 2025-11-03T16:37:00.8982656Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8982844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8982914Z res = mod(**inputs) 2025-11-03T16:37:00.8983175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8983262Z outputs = self.mobilebert( 2025-11-03T16:37:00.8983534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8983604Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8983877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8983961Z layer_outputs = layer_module( 2025-11-03T16:37:00.8984233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8984325Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8984592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8984716Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8984981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8985065Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8985069Z 2025-11-03T16:37:00.8985165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8985349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8985418Z res = mod(**inputs) 2025-11-03T16:37:00.8985682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8985770Z outputs = self.mobilebert( 2025-11-03T16:37:00.8986035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8986110Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8986376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8986442Z layer_outputs = layer_module( 2025-11-03T16:37:00.8986714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8986801Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8987076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8987193Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8987476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8987598Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8987867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8987959Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8987962Z 2025-11-03T16:37:00.8988059Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8988252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8988311Z res = mod(**inputs) 2025-11-03T16:37:00.8988589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8988662Z outputs = self.mobilebert( 2025-11-03T16:37:00.8988920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8988994Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8989265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8989332Z layer_outputs = layer_module( 2025-11-03T16:37:00.8989601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8989686Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8989965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8990067Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8990333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.8990409Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.8990413Z 2025-11-03T16:37:00.8990506Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8990699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8990756Z res = mod(**inputs) 2025-11-03T16:37:00.8991020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8991084Z outputs = self.mobilebert( 2025-11-03T16:37:00.8991346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8991419Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8991679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8991781Z layer_outputs = layer_module( 2025-11-03T16:37:00.8992038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8992129Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8992394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.8992493Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.8992756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.8992856Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.8992860Z 2025-11-03T16:37:00.8992960Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8993155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8993219Z res = mod(**inputs) 2025-11-03T16:37:00.8993475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8993542Z outputs = self.mobilebert( 2025-11-03T16:37:00.8993806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8993871Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8994199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8994274Z layer_outputs = layer_module( 2025-11-03T16:37:00.8994546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8994650Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8994931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8995055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8995338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.8995426Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.8995430Z 2025-11-03T16:37:00.8995537Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8995743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8995813Z res = mod(**inputs) 2025-11-03T16:37:00.8996071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8996149Z outputs = self.mobilebert( 2025-11-03T16:37:00.8996406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8996472Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8996740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8996806Z layer_outputs = layer_module( 2025-11-03T16:37:00.8997078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.8997163Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.8997430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.8997544Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.8997815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.8997935Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.8998196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.8998288Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.8998291Z 2025-11-03T16:37:00.8998386Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.8998566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.8998634Z res = mod(**inputs) 2025-11-03T16:37:00.8998895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.8998987Z outputs = self.mobilebert( 2025-11-03T16:37:00.8999243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.8999316Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.8999574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.8999640Z layer_outputs = layer_module( 2025-11-03T16:37:00.8999904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9000019Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9000282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9000358Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9000363Z 2025-11-03T16:37:00.9000456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9000645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9000703Z res = mod(**inputs) 2025-11-03T16:37:00.9000984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9001049Z outputs = self.mobilebert( 2025-11-03T16:37:00.9001315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9001380Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9001657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9001732Z layer_outputs = layer_module( 2025-11-03T16:37:00.9001990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9002107Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9002365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9002466Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9002476Z 2025-11-03T16:37:00.9002570Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9002750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9002815Z res = mod(**inputs) 2025-11-03T16:37:00.9003075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9003147Z outputs = self.mobilebert( 2025-11-03T16:37:00.9003419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9003486Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9003751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9003816Z layer_outputs = layer_module( 2025-11-03T16:37:00.9004080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9004224Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9004482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9004578Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9004581Z 2025-11-03T16:37:00.9004691Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9004879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9004938Z res = mod(**inputs) 2025-11-03T16:37:00.9005203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9005269Z outputs = self.mobilebert( 2025-11-03T16:37:00.9005525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9005596Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9005854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9005925Z layer_outputs = layer_module( 2025-11-03T16:37:00.9006181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9006327Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9006589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9006720Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9006985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9007068Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9007072Z 2025-11-03T16:37:00.9007171Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9007369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9007429Z res = mod(**inputs) 2025-11-03T16:37:00.9007692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9007758Z outputs = self.mobilebert( 2025-11-03T16:37:00.9008019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9008086Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9008345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9008417Z layer_outputs = layer_module( 2025-11-03T16:37:00.9008675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9008827Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9009082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9009221Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9009479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9009557Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9009568Z 2025-11-03T16:37:00.9009661Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9009843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9009909Z res = mod(**inputs) 2025-11-03T16:37:00.9010166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9010237Z outputs = self.mobilebert( 2025-11-03T16:37:00.9010494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9010578Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9010844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9010908Z layer_outputs = layer_module( 2025-11-03T16:37:00.9011171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9011313Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9011568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9011693Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9011949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9012074Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9012334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9012440Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9012443Z 2025-11-03T16:37:00.9012540Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9012721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9012788Z res = mod(**inputs) 2025-11-03T16:37:00.9013082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9013156Z outputs = self.mobilebert( 2025-11-03T16:37:00.9013593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9013671Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9013937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9014003Z layer_outputs = layer_module( 2025-11-03T16:37:00.9014273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9014420Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9014696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9014801Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9015069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9015198Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9015202Z 2025-11-03T16:37:00.9015297Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9015491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9015552Z res = mod(**inputs) 2025-11-03T16:37:00.9015832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9015898Z outputs = self.mobilebert( 2025-11-03T16:37:00.9016170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9016248Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9016522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9016622Z layer_outputs = layer_module( 2025-11-03T16:37:00.9016942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9017026Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9017304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9017373Z self_outputs = self.self( 2025-11-03T16:37:00.9017646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9017715Z self.value(value_tensor) 2025-11-03T16:37:00.9017718Z 2025-11-03T16:37:00.9017823Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9018012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9018076Z res = mod(**inputs) 2025-11-03T16:37:00.9018347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9018411Z outputs = self.mobilebert( 2025-11-03T16:37:00.9018706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9018775Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9019047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9019120Z layer_outputs = layer_module( 2025-11-03T16:37:00.9019415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9019574Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9019842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9019952Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9020218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9020295Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9020299Z 2025-11-03T16:37:00.9020405Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9020591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9020658Z res = mod(**inputs) 2025-11-03T16:37:00.9020927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9020993Z outputs = self.mobilebert( 2025-11-03T16:37:00.9021264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9021348Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9021619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9021686Z layer_outputs = layer_module( 2025-11-03T16:37:00.9021954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9022105Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9022370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9022482Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9022759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9022849Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9023121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9023207Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9023217Z 2025-11-03T16:37:00.9023312Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9023495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9023563Z res = mod(**inputs) 2025-11-03T16:37:00.9023827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9023900Z outputs = self.mobilebert( 2025-11-03T16:37:00.9024164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9024236Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9024507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9024592Z layer_outputs = layer_module( 2025-11-03T16:37:00.9024864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9024946Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9025207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9025298Z self_outputs = self.self( 2025-11-03T16:37:00.9025562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9025640Z self.query(query_tensor) 2025-11-03T16:37:00.9025643Z 2025-11-03T16:37:00.9025740Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9025933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9025994Z res = mod(**inputs) 2025-11-03T16:37:00.9026258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9026334Z outputs = self.mobilebert( 2025-11-03T16:37:00.9026595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9026670Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9026935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9027005Z layer_outputs = layer_module( 2025-11-03T16:37:00.9027292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9027374Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9027646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9027711Z self_outputs = self.self( 2025-11-03T16:37:00.9027980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9028044Z self.key(key_tensor) 2025-11-03T16:37:00.9028047Z 2025-11-03T16:37:00.9028126Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9028212Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9028307Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9028499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9028577Z res = mod(**inputs) 2025-11-03T16:37:00.9028840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9028914Z outputs = self.mobilebert( 2025-11-03T16:37:00.9029182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9029258Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9029522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9029586Z layer_outputs = layer_module( 2025-11-03T16:37:00.9029867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9029943Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9030211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9030324Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9030607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9030684Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9030687Z 2025-11-03T16:37:00.9030780Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9030966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9031023Z res = mod(**inputs) 2025-11-03T16:37:00.9031303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9031370Z outputs = self.mobilebert( 2025-11-03T16:37:00.9031631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9031703Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9031963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9032034Z layer_outputs = layer_module( 2025-11-03T16:37:00.9032291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9032376Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9032633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9032744Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9033010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9033144Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9033406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9033490Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9033493Z 2025-11-03T16:37:00.9033594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9033774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9033832Z res = mod(**inputs) 2025-11-03T16:37:00.9034167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9034242Z outputs = self.mobilebert( 2025-11-03T16:37:00.9034547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9034619Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9034912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9034988Z layer_outputs = layer_module( 2025-11-03T16:37:00.9035257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9035354Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9035629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9035733Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9035999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9036080Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9036083Z 2025-11-03T16:37:00.9036187Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9036371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9036462Z res = mod(**inputs) 2025-11-03T16:37:00.9036732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9036798Z outputs = self.mobilebert( 2025-11-03T16:37:00.9037067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9037149Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9037417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9037483Z layer_outputs = layer_module( 2025-11-03T16:37:00.9037741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9037837Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9038095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9038203Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9038461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9038570Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9038573Z 2025-11-03T16:37:00.9038666Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9038849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9038933Z res = mod(**inputs) 2025-11-03T16:37:00.9039189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9039261Z outputs = self.mobilebert( 2025-11-03T16:37:00.9039519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9039585Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9039850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9039914Z layer_outputs = layer_module( 2025-11-03T16:37:00.9040179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9040280Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9040548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9040662Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9040922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9041007Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9041010Z 2025-11-03T16:37:00.9041103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9041291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9041351Z res = mod(**inputs) 2025-11-03T16:37:00.9041617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9041683Z outputs = self.mobilebert( 2025-11-03T16:37:00.9041942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9042015Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9042289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9042363Z layer_outputs = layer_module( 2025-11-03T16:37:00.9042625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9042711Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9042993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9043110Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9043379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9043492Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9043760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9043845Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9043848Z 2025-11-03T16:37:00.9043943Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9044134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9044194Z res = mod(**inputs) 2025-11-03T16:37:00.9044460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9044524Z outputs = self.mobilebert( 2025-11-03T16:37:00.9044798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9044873Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9045133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9045204Z layer_outputs = layer_module( 2025-11-03T16:37:00.9045460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9045551Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9045809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9045910Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9046175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9046270Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9046273Z 2025-11-03T16:37:00.9046373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9046556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9046615Z res = mod(**inputs) 2025-11-03T16:37:00.9046880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9046944Z outputs = self.mobilebert( 2025-11-03T16:37:00.9047210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9047276Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9047540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9047607Z layer_outputs = layer_module( 2025-11-03T16:37:00.9047864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9047958Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9048234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9048345Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9048602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9048719Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9048729Z 2025-11-03T16:37:00.9048825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9049008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9049076Z res = mod(**inputs) 2025-11-03T16:37:00.9049336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9049408Z outputs = self.mobilebert( 2025-11-03T16:37:00.9049665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9049728Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9049992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9050058Z layer_outputs = layer_module( 2025-11-03T16:37:00.9050322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9050425Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9050684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9050806Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9051064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9051149Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9051152Z 2025-11-03T16:37:00.9051247Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9051434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9051494Z res = mod(**inputs) 2025-11-03T16:37:00.9051757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9051846Z outputs = self.mobilebert( 2025-11-03T16:37:00.9052103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9052176Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9052434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9052499Z layer_outputs = layer_module( 2025-11-03T16:37:00.9052764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9052850Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9053117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9053230Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9053496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9053610Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9053888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9053980Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9053983Z 2025-11-03T16:37:00.9054077Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9054264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9054321Z res = mod(**inputs) 2025-11-03T16:37:00.9054605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9054671Z outputs = self.mobilebert( 2025-11-03T16:37:00.9054928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9055001Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9055260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9055331Z layer_outputs = layer_module( 2025-11-03T16:37:00.9055588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9055673Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9055942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9056045Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9056316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9056409Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9056412Z 2025-11-03T16:37:00.9056513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9056696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9056755Z res = mod(**inputs) 2025-11-03T16:37:00.9057020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9057085Z outputs = self.mobilebert( 2025-11-03T16:37:00.9057352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9057417Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9057677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9057774Z layer_outputs = layer_module( 2025-11-03T16:37:00.9058032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9058126Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9058382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9058491Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9058749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9058850Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9058854Z 2025-11-03T16:37:00.9058955Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9059138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9059204Z res = mod(**inputs) 2025-11-03T16:37:00.9059461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9059550Z outputs = self.mobilebert( 2025-11-03T16:37:00.9059818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9059883Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9060148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9060227Z layer_outputs = layer_module( 2025-11-03T16:37:00.9060496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9060581Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9060838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9060957Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9061216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9061300Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9061303Z 2025-11-03T16:37:00.9061397Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9061581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9061646Z res = mod(**inputs) 2025-11-03T16:37:00.9061903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9061993Z outputs = self.mobilebert( 2025-11-03T16:37:00.9062252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9062323Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9062585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9062649Z layer_outputs = layer_module( 2025-11-03T16:37:00.9062915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9062999Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9063263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9063396Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9063654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9063772Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9064031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9064120Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9064123Z 2025-11-03T16:37:00.9064216Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9064403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9064462Z res = mod(**inputs) 2025-11-03T16:37:00.9064717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9064791Z outputs = self.mobilebert( 2025-11-03T16:37:00.9065049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9065122Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9065394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9065462Z layer_outputs = layer_module( 2025-11-03T16:37:00.9065726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9065838Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9066121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9066199Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9066204Z 2025-11-03T16:37:00.9066303Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9066484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9066541Z res = mod(**inputs) 2025-11-03T16:37:00.9066807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9066870Z outputs = self.mobilebert( 2025-11-03T16:37:00.9067133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9067198Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9067459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9067529Z layer_outputs = layer_module( 2025-11-03T16:37:00.9067789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9067924Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9068185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9068296Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9068300Z 2025-11-03T16:37:00.9068394Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9068576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9068642Z res = mod(**inputs) 2025-11-03T16:37:00.9068905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9068974Z outputs = self.mobilebert( 2025-11-03T16:37:00.9069277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9069344Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9069663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9069731Z layer_outputs = layer_module( 2025-11-03T16:37:00.9070009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9070160Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9070435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9070526Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9070530Z 2025-11-03T16:37:00.9070627Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9070826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9070885Z res = mod(**inputs) 2025-11-03T16:37:00.9071176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9071242Z outputs = self.mobilebert( 2025-11-03T16:37:00.9071514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9071584Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9071908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9071987Z layer_outputs = layer_module( 2025-11-03T16:37:00.9072253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9072412Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9072677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9072794Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9073069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9073155Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9073158Z 2025-11-03T16:37:00.9073260Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9073449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9073515Z res = mod(**inputs) 2025-11-03T16:37:00.9073780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9073862Z outputs = self.mobilebert( 2025-11-03T16:37:00.9074211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9074287Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9074571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9074639Z layer_outputs = layer_module( 2025-11-03T16:37:00.9074917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9075081Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9075376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9075526Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9075798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9075889Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9075892Z 2025-11-03T16:37:00.9075990Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9076178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9076249Z res = mod(**inputs) 2025-11-03T16:37:00.9076521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9076595Z outputs = self.mobilebert( 2025-11-03T16:37:00.9076867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9076937Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9077216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9077282Z layer_outputs = layer_module( 2025-11-03T16:37:00.9077581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9077730Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9078008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9078139Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9078414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9078543Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9078822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9078919Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9078922Z 2025-11-03T16:37:00.9079021Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9079218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9079278Z res = mod(**inputs) 2025-11-03T16:37:00.9079551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9079626Z outputs = self.mobilebert( 2025-11-03T16:37:00.9079900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9080015Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9080283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9080347Z layer_outputs = layer_module( 2025-11-03T16:37:00.9080620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9080771Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9081044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9081146Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9081417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9081510Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9081514Z 2025-11-03T16:37:00.9081608Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9081801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9081862Z res = mod(**inputs) 2025-11-03T16:37:00.9082130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9082196Z outputs = self.mobilebert( 2025-11-03T16:37:00.9082459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9082534Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9082797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9082879Z layer_outputs = layer_module( 2025-11-03T16:37:00.9083144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9083231Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9083510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9083578Z self_outputs = self.self( 2025-11-03T16:37:00.9083863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9083929Z self.value(value_tensor) 2025-11-03T16:37:00.9083932Z 2025-11-03T16:37:00.9084050Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9084236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9084297Z res = mod(**inputs) 2025-11-03T16:37:00.9084565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9084630Z outputs = self.mobilebert( 2025-11-03T16:37:00.9084897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9084963Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9085234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9085300Z layer_outputs = layer_module( 2025-11-03T16:37:00.9085564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9085722Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9085989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9086116Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9086382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9086460Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9086471Z 2025-11-03T16:37:00.9086568Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9086752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9086819Z res = mod(**inputs) 2025-11-03T16:37:00.9087082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9087155Z outputs = self.mobilebert( 2025-11-03T16:37:00.9087441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9087511Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9087789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9087856Z layer_outputs = layer_module( 2025-11-03T16:37:00.9088130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9088280Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9088550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9088659Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9088930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9089021Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9089298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9089400Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9089403Z 2025-11-03T16:37:00.9089498Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9089678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9089744Z res = mod(**inputs) 2025-11-03T16:37:00.9090014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9090087Z outputs = self.mobilebert( 2025-11-03T16:37:00.9090349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9090418Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9090686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9090751Z layer_outputs = layer_module( 2025-11-03T16:37:00.9091014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9091093Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9091357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9091423Z self_outputs = self.self( 2025-11-03T16:37:00.9091685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9091777Z self.query(query_tensor) 2025-11-03T16:37:00.9091780Z 2025-11-03T16:37:00.9091877Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9092063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9092123Z res = mod(**inputs) 2025-11-03T16:37:00.9092384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9092456Z outputs = self.mobilebert( 2025-11-03T16:37:00.9092713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9092785Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9093045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9093131Z layer_outputs = layer_module( 2025-11-03T16:37:00.9093391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9093468Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9093732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9093798Z self_outputs = self.self( 2025-11-03T16:37:00.9094061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9094123Z self.key(key_tensor) 2025-11-03T16:37:00.9094126Z 2025-11-03T16:37:00.9094201Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9094283Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9094377Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9094566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9094626Z res = mod(**inputs) 2025-11-03T16:37:00.9094887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9094959Z outputs = self.mobilebert( 2025-11-03T16:37:00.9095228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9095306Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9095570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9095642Z layer_outputs = layer_module( 2025-11-03T16:37:00.9095912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9095989Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9096257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9096369Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9096633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9096709Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9096712Z 2025-11-03T16:37:00.9096811Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9096991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9097048Z res = mod(**inputs) 2025-11-03T16:37:00.9097312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9097378Z outputs = self.mobilebert( 2025-11-03T16:37:00.9097655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9097722Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9097979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9098050Z layer_outputs = layer_module( 2025-11-03T16:37:00.9098306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9098386Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9098643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9098754Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9099035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9099153Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9099418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9099503Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9099506Z 2025-11-03T16:37:00.9099607Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9099790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9099846Z res = mod(**inputs) 2025-11-03T16:37:00.9100113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9100177Z outputs = self.mobilebert( 2025-11-03T16:37:00.9100443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9100509Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9100766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9100860Z layer_outputs = layer_module( 2025-11-03T16:37:00.9101118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9101214Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9101470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9101603Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9101862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9101942Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9101945Z 2025-11-03T16:37:00.9102047Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9102230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9102297Z res = mod(**inputs) 2025-11-03T16:37:00.9102552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9102624Z outputs = self.mobilebert( 2025-11-03T16:37:00.9102881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9102948Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9103212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9103296Z layer_outputs = layer_module( 2025-11-03T16:37:00.9103562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9103651Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9103909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9104019Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9104278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9104390Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9104394Z 2025-11-03T16:37:00.9104488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9104675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9104754Z res = mod(**inputs) 2025-11-03T16:37:00.9105011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9105084Z outputs = self.mobilebert( 2025-11-03T16:37:00.9105340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9105416Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9105672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9105735Z layer_outputs = layer_module( 2025-11-03T16:37:00.9106004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9106092Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9106358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9106473Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9106762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9106840Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9106843Z 2025-11-03T16:37:00.9106934Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9107127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9107185Z res = mod(**inputs) 2025-11-03T16:37:00.9107462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9107529Z outputs = self.mobilebert( 2025-11-03T16:37:00.9107807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9107882Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9108152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9108225Z layer_outputs = layer_module( 2025-11-03T16:37:00.9108492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9108588Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9108863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9108979Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9109276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9109391Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9109669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9109755Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9109758Z 2025-11-03T16:37:00.9109860Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9110048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9110107Z res = mod(**inputs) 2025-11-03T16:37:00.9110378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9110445Z outputs = self.mobilebert( 2025-11-03T16:37:00.9110734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9110802Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9111067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9111141Z layer_outputs = layer_module( 2025-11-03T16:37:00.9111405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9111500Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9111767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9111872Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9112142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9112223Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9112226Z 2025-11-03T16:37:00.9112328Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9112531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9112599Z res = mod(**inputs) 2025-11-03T16:37:00.9112862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9112927Z outputs = self.mobilebert( 2025-11-03T16:37:00.9113389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9113471Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9113753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9113823Z layer_outputs = layer_module( 2025-11-03T16:37:00.9114147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9114260Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9114531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9114647Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9114917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9115036Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9115039Z 2025-11-03T16:37:00.9115138Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9115370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9115439Z res = mod(**inputs) 2025-11-03T16:37:00.9115702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9115776Z outputs = self.mobilebert( 2025-11-03T16:37:00.9116044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9116115Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9116388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9116456Z layer_outputs = layer_module( 2025-11-03T16:37:00.9116727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9116841Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9117111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9117229Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9117496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9117585Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9117589Z 2025-11-03T16:37:00.9117686Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9117883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9117943Z res = mod(**inputs) 2025-11-03T16:37:00.9118219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9118288Z outputs = self.mobilebert( 2025-11-03T16:37:00.9118555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9118630Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9118921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9118997Z layer_outputs = layer_module( 2025-11-03T16:37:00.9119261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9119351Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9119637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9119756Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9120041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9120155Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9120433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9120520Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9120523Z 2025-11-03T16:37:00.9120619Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9120813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9120872Z res = mod(**inputs) 2025-11-03T16:37:00.9121147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9121216Z outputs = self.mobilebert( 2025-11-03T16:37:00.9121502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9121578Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9121844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9121916Z layer_outputs = layer_module( 2025-11-03T16:37:00.9122179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9122275Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9122540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9122644Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9122933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9123014Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9123017Z 2025-11-03T16:37:00.9123120Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9123309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9123368Z res = mod(**inputs) 2025-11-03T16:37:00.9123638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9123705Z outputs = self.mobilebert( 2025-11-03T16:37:00.9126445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9126529Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9126801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9126880Z layer_outputs = layer_module( 2025-11-03T16:37:00.9127140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9127240Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9127501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9127608Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9127869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9128026Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9128030Z 2025-11-03T16:37:00.9128127Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9128319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9128378Z res = mod(**inputs) 2025-11-03T16:37:00.9128641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9128707Z outputs = self.mobilebert( 2025-11-03T16:37:00.9128965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9129038Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9129298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9129370Z layer_outputs = layer_module( 2025-11-03T16:37:00.9129626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9129729Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9129995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9130110Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9130375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9130451Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9130455Z 2025-11-03T16:37:00.9130554Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9130740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9130798Z res = mod(**inputs) 2025-11-03T16:37:00.9131062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9131151Z outputs = self.mobilebert( 2025-11-03T16:37:00.9131411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9131477Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9131735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9131808Z layer_outputs = layer_module( 2025-11-03T16:37:00.9132063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9132156Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9132471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9132595Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9132851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9132960Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9133224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9133309Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9133313Z 2025-11-03T16:37:00.9133412Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9133592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9133674Z res = mod(**inputs) 2025-11-03T16:37:00.9133933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9133999Z outputs = self.mobilebert( 2025-11-03T16:37:00.9134263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9134328Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9134590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9134656Z layer_outputs = layer_module( 2025-11-03T16:37:00.9134911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9135029Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9135289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9135371Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9135392Z 2025-11-03T16:37:00.9135487Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9135676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9135736Z res = mod(**inputs) 2025-11-03T16:37:00.9135992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9136065Z outputs = self.mobilebert( 2025-11-03T16:37:00.9136325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9136400Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9136664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9136728Z layer_outputs = layer_module( 2025-11-03T16:37:00.9137010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9137121Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9137385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9137487Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9137491Z 2025-11-03T16:37:00.9137593Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9137773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9137832Z res = mod(**inputs) 2025-11-03T16:37:00.9138121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9138189Z outputs = self.mobilebert( 2025-11-03T16:37:00.9138451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9138519Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9138776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9138846Z layer_outputs = layer_module( 2025-11-03T16:37:00.9139104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9139255Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9139529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9139626Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9139629Z 2025-11-03T16:37:00.9139724Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9139906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9139970Z res = mod(**inputs) 2025-11-03T16:37:00.9140227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9140298Z outputs = self.mobilebert( 2025-11-03T16:37:00.9140553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9140620Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9140888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9140954Z layer_outputs = layer_module( 2025-11-03T16:37:00.9141215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9141379Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9141642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9141757Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9142011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9142103Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9142106Z 2025-11-03T16:37:00.9142198Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9142387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9142447Z res = mod(**inputs) 2025-11-03T16:37:00.9142723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9142795Z outputs = self.mobilebert( 2025-11-03T16:37:00.9143055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9143129Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9143388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9143460Z layer_outputs = layer_module( 2025-11-03T16:37:00.9143721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9143879Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9144147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9144262Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9144531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9144607Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9144611Z 2025-11-03T16:37:00.9144710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9144891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9144948Z res = mod(**inputs) 2025-11-03T16:37:00.9145224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9145291Z outputs = self.mobilebert( 2025-11-03T16:37:00.9145555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9145622Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9145879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9145950Z layer_outputs = layer_module( 2025-11-03T16:37:00.9146212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9146361Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9146618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9146737Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9146994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9147123Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9147391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9147473Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9147476Z 2025-11-03T16:37:00.9147577Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9147758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9147818Z res = mod(**inputs) 2025-11-03T16:37:00.9148085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9148152Z outputs = self.mobilebert( 2025-11-03T16:37:00.9148418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9148505Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9148769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9148833Z layer_outputs = layer_module( 2025-11-03T16:37:00.9149087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9149239Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9149496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9149603Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9149874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9149955Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9149967Z 2025-11-03T16:37:00.9150061Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9150242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9150308Z res = mod(**inputs) 2025-11-03T16:37:00.9150565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9150636Z outputs = self.mobilebert( 2025-11-03T16:37:00.9150893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9150972Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9151244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9151310Z layer_outputs = layer_module( 2025-11-03T16:37:00.9151575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9151655Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9151911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9151986Z self_outputs = self.self( 2025-11-03T16:37:00.9152244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9152316Z self.value(value_tensor) 2025-11-03T16:37:00.9152319Z 2025-11-03T16:37:00.9152416Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9152606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9152667Z res = mod(**inputs) 2025-11-03T16:37:00.9152943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9153015Z outputs = self.mobilebert( 2025-11-03T16:37:00.9153272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9153346Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9153605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9153669Z layer_outputs = layer_module( 2025-11-03T16:37:00.9153940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9154167Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9154469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9154577Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9154849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9154929Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9154933Z 2025-11-03T16:37:00.9155029Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9155227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9155287Z res = mod(**inputs) 2025-11-03T16:37:00.9155578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9155646Z outputs = self.mobilebert( 2025-11-03T16:37:00.9155914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9155993Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9156261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9156337Z layer_outputs = layer_module( 2025-11-03T16:37:00.9156603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9156763Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9157044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9157152Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9157426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9157511Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9157785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9157873Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9157876Z 2025-11-03T16:37:00.9157979Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9158167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9158227Z res = mod(**inputs) 2025-11-03T16:37:00.9158500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9158569Z outputs = self.mobilebert( 2025-11-03T16:37:00.9158838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9158924Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9159189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9159263Z layer_outputs = layer_module( 2025-11-03T16:37:00.9159528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9159617Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9159890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9159958Z self_outputs = self.self( 2025-11-03T16:37:00.9160235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9160321Z self.query(query_tensor) 2025-11-03T16:37:00.9160325Z 2025-11-03T16:37:00.9160429Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9160619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9160688Z res = mod(**inputs) 2025-11-03T16:37:00.9160957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9161024Z outputs = self.mobilebert( 2025-11-03T16:37:00.9161302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9161370Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9161658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9161724Z layer_outputs = layer_module( 2025-11-03T16:37:00.9161995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9162080Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9162348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9162418Z self_outputs = self.self( 2025-11-03T16:37:00.9162683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9162758Z self.key(key_tensor) 2025-11-03T16:37:00.9162761Z 2025-11-03T16:37:00.9162837Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9162931Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9163037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9163223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9163293Z res = mod(**inputs) 2025-11-03T16:37:00.9163559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9163625Z outputs = self.mobilebert( 2025-11-03T16:37:00.9163917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9163985Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9164257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9164324Z layer_outputs = layer_module( 2025-11-03T16:37:00.9164591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9164678Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9164993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9165117Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9165389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9165476Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9165479Z 2025-11-03T16:37:00.9165575Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9165764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9165833Z res = mod(**inputs) 2025-11-03T16:37:00.9166107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9166179Z outputs = self.mobilebert( 2025-11-03T16:37:00.9166459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9166527Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9166809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9166878Z layer_outputs = layer_module( 2025-11-03T16:37:00.9167147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9167225Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9167497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9167632Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9167902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9168031Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9168298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9168390Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9168393Z 2025-11-03T16:37:00.9168488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9168683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9168744Z res = mod(**inputs) 2025-11-03T16:37:00.9169029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9169106Z outputs = self.mobilebert( 2025-11-03T16:37:00.9169374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9169449Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9169713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9169778Z layer_outputs = layer_module( 2025-11-03T16:37:00.9170053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9170142Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9170418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9170525Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9170797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9170903Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9170907Z 2025-11-03T16:37:00.9171003Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9171200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9171259Z res = mod(**inputs) 2025-11-03T16:37:00.9171535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9171600Z outputs = self.mobilebert( 2025-11-03T16:37:00.9171867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9171944Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9172213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9172307Z layer_outputs = layer_module( 2025-11-03T16:37:00.9172573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9172668Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9172934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9173040Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9173316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9173426Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9173429Z 2025-11-03T16:37:00.9173558Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9173745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9173806Z res = mod(**inputs) 2025-11-03T16:37:00.9174083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9174147Z outputs = self.mobilebert( 2025-11-03T16:37:00.9174420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9174487Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9174755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9174823Z layer_outputs = layer_module( 2025-11-03T16:37:00.9175102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9175200Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9175469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9175592Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9175859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9175938Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9175948Z 2025-11-03T16:37:00.9176044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9176227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9176294Z res = mod(**inputs) 2025-11-03T16:37:00.9176564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9176639Z outputs = self.mobilebert( 2025-11-03T16:37:00.9176921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9176987Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9177256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9177322Z layer_outputs = layer_module( 2025-11-03T16:37:00.9177591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9177679Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9177945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9178067Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9178351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9178472Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9178737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9178829Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9178832Z 2025-11-03T16:37:00.9178928Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9179114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9179181Z res = mod(**inputs) 2025-11-03T16:37:00.9179461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9179537Z outputs = self.mobilebert( 2025-11-03T16:37:00.9179812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9179881Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9180154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9180220Z layer_outputs = layer_module( 2025-11-03T16:37:00.9180493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9180581Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9180869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9180976Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9181240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9181329Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9181332Z 2025-11-03T16:37:00.9181427Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9181621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9181693Z res = mod(**inputs) 2025-11-03T16:37:00.9181958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9182024Z outputs = self.mobilebert( 2025-11-03T16:37:00.9182300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9182371Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9182650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9182736Z layer_outputs = layer_module( 2025-11-03T16:37:00.9183006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9183102Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9183372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9183484Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9183766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9183881Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9183886Z 2025-11-03T16:37:00.9183984Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9184189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9184257Z res = mod(**inputs) 2025-11-03T16:37:00.9184528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9184603Z outputs = self.mobilebert( 2025-11-03T16:37:00.9184875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9184943Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9185223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9185289Z layer_outputs = layer_module( 2025-11-03T16:37:00.9185589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9185679Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9185952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9186071Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9186335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9186423Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9186426Z 2025-11-03T16:37:00.9186522Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9186714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9186790Z res = mod(**inputs) 2025-11-03T16:37:00.9187065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9187139Z outputs = self.mobilebert( 2025-11-03T16:37:00.9187393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9187466Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9187721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9187793Z layer_outputs = layer_module( 2025-11-03T16:37:00.9188049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9188133Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9188403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9188518Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9188839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9188956Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9189233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9189325Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9189328Z 2025-11-03T16:37:00.9189424Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9189617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9189678Z res = mod(**inputs) 2025-11-03T16:37:00.9189948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9190032Z outputs = self.mobilebert( 2025-11-03T16:37:00.9190299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9190374Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9190644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9190719Z layer_outputs = layer_module( 2025-11-03T16:37:00.9190985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9191073Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9191367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9191473Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9191750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9191829Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9191833Z 2025-11-03T16:37:00.9191935Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9192123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9192182Z res = mod(**inputs) 2025-11-03T16:37:00.9192452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9192517Z outputs = self.mobilebert( 2025-11-03T16:37:00.9192812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9192883Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9193147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9193222Z layer_outputs = layer_module( 2025-11-03T16:37:00.9193484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9193582Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9193844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9193956Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9194290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9194405Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9194408Z 2025-11-03T16:37:00.9194515Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9194725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9194796Z res = mod(**inputs) 2025-11-03T16:37:00.9195061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9195127Z outputs = self.mobilebert( 2025-11-03T16:37:00.9195396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9195464Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9195746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9195812Z layer_outputs = layer_module( 2025-11-03T16:37:00.9196078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9196184Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9196445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9196571Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9196827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9196911Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9196914Z 2025-11-03T16:37:00.9197007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9197194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9197266Z res = mod(**inputs) 2025-11-03T16:37:00.9197525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9197600Z outputs = self.mobilebert( 2025-11-03T16:37:00.9197857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9197930Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9198188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9198251Z layer_outputs = layer_module( 2025-11-03T16:37:00.9198517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9198602Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9198885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9199001Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9199267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9199380Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9199637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9199729Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9199732Z 2025-11-03T16:37:00.9199824Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9200010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9200072Z res = mod(**inputs) 2025-11-03T16:37:00.9200331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9200422Z outputs = self.mobilebert( 2025-11-03T16:37:00.9200686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9200759Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9201023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9201096Z layer_outputs = layer_module( 2025-11-03T16:37:00.9201359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9201473Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9201746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9201823Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9201843Z 2025-11-03T16:37:00.9201946Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9202128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9202185Z res = mod(**inputs) 2025-11-03T16:37:00.9202452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9202516Z outputs = self.mobilebert( 2025-11-03T16:37:00.9202780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9202845Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9203126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9203192Z layer_outputs = layer_module( 2025-11-03T16:37:00.9203451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9203570Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9203825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9203935Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9203938Z 2025-11-03T16:37:00.9204031Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9204211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9204276Z res = mod(**inputs) 2025-11-03T16:37:00.9204550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9204625Z outputs = self.mobilebert( 2025-11-03T16:37:00.9204896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9204968Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9205227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9205291Z layer_outputs = layer_module( 2025-11-03T16:37:00.9205557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9205700Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9205965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9206054Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9206058Z 2025-11-03T16:37:00.9206161Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9206356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9206416Z res = mod(**inputs) 2025-11-03T16:37:00.9206680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9206744Z outputs = self.mobilebert( 2025-11-03T16:37:00.9207009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9207074Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9207343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9207418Z layer_outputs = layer_module( 2025-11-03T16:37:00.9207677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9207848Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9208106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9208219Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9208483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9208567Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9208571Z 2025-11-03T16:37:00.9208674Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9208874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9208941Z res = mod(**inputs) 2025-11-03T16:37:00.9209201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9209269Z outputs = self.mobilebert( 2025-11-03T16:37:00.9209538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9209605Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9209874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9209937Z layer_outputs = layer_module( 2025-11-03T16:37:00.9210198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9210368Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9210628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9210754Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9211010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9211096Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9211099Z 2025-11-03T16:37:00.9211193Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9211372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9211439Z res = mod(**inputs) 2025-11-03T16:37:00.9211695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9211769Z outputs = self.mobilebert( 2025-11-03T16:37:00.9212026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9212121Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9212386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9212450Z layer_outputs = layer_module( 2025-11-03T16:37:00.9212723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9212863Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9213136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9213504Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9213780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9213946Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9214201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9214296Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9214299Z 2025-11-03T16:37:00.9214395Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9214583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9214642Z res = mod(**inputs) 2025-11-03T16:37:00.9214898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9215007Z outputs = self.mobilebert( 2025-11-03T16:37:00.9215273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9215352Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9215612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9215676Z layer_outputs = layer_module( 2025-11-03T16:37:00.9215942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9216091Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9216358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9216483Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9216748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9216826Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9216830Z 2025-11-03T16:37:00.9216923Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9217112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9217171Z res = mod(**inputs) 2025-11-03T16:37:00.9217433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9217496Z outputs = self.mobilebert( 2025-11-03T16:37:00.9217757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9217824Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9218079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9218177Z layer_outputs = layer_module( 2025-11-03T16:37:00.9218434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9218518Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9218773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9218841Z self_outputs = self.self( 2025-11-03T16:37:00.9219107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9219171Z self.value(value_tensor) 2025-11-03T16:37:00.9219175Z 2025-11-03T16:37:00.9219277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9219457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9219529Z res = mod(**inputs) 2025-11-03T16:37:00.9219793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9219857Z outputs = self.mobilebert( 2025-11-03T16:37:00.9220120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9220185Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9220448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9220512Z layer_outputs = layer_module( 2025-11-03T16:37:00.9220767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9220937Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9221196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9221308Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9221563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9221647Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9221650Z 2025-11-03T16:37:00.9221742Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9221922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9221987Z res = mod(**inputs) 2025-11-03T16:37:00.9222258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9222330Z outputs = self.mobilebert( 2025-11-03T16:37:00.9222585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9222653Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9222919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9222983Z layer_outputs = layer_module( 2025-11-03T16:37:00.9223247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9223393Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9223657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9223760Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9224016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9224123Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9224379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9224471Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9224474Z 2025-11-03T16:37:00.9224570Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9224747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9224812Z res = mod(**inputs) 2025-11-03T16:37:00.9225068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9225142Z outputs = self.mobilebert( 2025-11-03T16:37:00.9225398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9225490Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9225747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9225811Z layer_outputs = layer_module( 2025-11-03T16:37:00.9226078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9226157Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9226424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9226490Z self_outputs = self.self( 2025-11-03T16:37:00.9226784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9226861Z self.query(query_tensor) 2025-11-03T16:37:00.9226864Z 2025-11-03T16:37:00.9226959Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9227149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9227209Z res = mod(**inputs) 2025-11-03T16:37:00.9227472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9227535Z outputs = self.mobilebert( 2025-11-03T16:37:00.9227794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9227868Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9228144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9228215Z layer_outputs = layer_module( 2025-11-03T16:37:00.9228481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9228558Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9228822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9228885Z self_outputs = self.self( 2025-11-03T16:37:00.9229148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9229211Z self.key(key_tensor) 2025-11-03T16:37:00.9229214Z 2025-11-03T16:37:00.9229294Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9229368Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9229463Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9229654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9229732Z res = mod(**inputs) 2025-11-03T16:37:00.9230007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9230071Z outputs = self.mobilebert( 2025-11-03T16:37:00.9230342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9230417Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9230687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9230759Z layer_outputs = layer_module( 2025-11-03T16:37:00.9231030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9231110Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9231398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9231513Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9231785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9231865Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9231868Z 2025-11-03T16:37:00.9231969Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9232158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9232218Z res = mod(**inputs) 2025-11-03T16:37:00.9232506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9232572Z outputs = self.mobilebert( 2025-11-03T16:37:00.9232843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9232913Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9233176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9233249Z layer_outputs = layer_module( 2025-11-03T16:37:00.9233514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9233601Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9233885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9234066Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9234347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9234470Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9234748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9234835Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9234838Z 2025-11-03T16:37:00.9234941Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9235138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9235197Z res = mod(**inputs) 2025-11-03T16:37:00.9235468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9235533Z outputs = self.mobilebert( 2025-11-03T16:37:00.9235807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9235901Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9236181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9236248Z layer_outputs = layer_module( 2025-11-03T16:37:00.9236517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9236616Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9236887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9237003Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9237276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9237383Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9237395Z 2025-11-03T16:37:00.9237493Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9237680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9237747Z res = mod(**inputs) 2025-11-03T16:37:00.9238013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9238086Z outputs = self.mobilebert( 2025-11-03T16:37:00.9238351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9238419Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9238711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9238782Z layer_outputs = layer_module( 2025-11-03T16:37:00.9239056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9239146Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9239420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9239531Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9239798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9239914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9239935Z 2025-11-03T16:37:00.9240036Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9240231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9240294Z res = mod(**inputs) 2025-11-03T16:37:00.9240562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9240638Z outputs = self.mobilebert( 2025-11-03T16:37:00.9240903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9240978Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9241240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9241306Z layer_outputs = layer_module( 2025-11-03T16:37:00.9241582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9241670Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9241965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9242084Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9242368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9242448Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9242451Z 2025-11-03T16:37:00.9242546Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9242738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9242796Z res = mod(**inputs) 2025-11-03T16:37:00.9243067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9243148Z outputs = self.mobilebert( 2025-11-03T16:37:00.9243412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9243485Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9243749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9243822Z layer_outputs = layer_module( 2025-11-03T16:37:00.9244087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9244182Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9244468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9244587Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9244861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9244978Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9245266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9245351Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9245355Z 2025-11-03T16:37:00.9245459Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9245647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9245707Z res = mod(**inputs) 2025-11-03T16:37:00.9245999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9246067Z outputs = self.mobilebert( 2025-11-03T16:37:00.9246347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9246413Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9246672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9246744Z layer_outputs = layer_module( 2025-11-03T16:37:00.9247003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9247095Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9247350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9247461Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9247716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9247814Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9247818Z 2025-11-03T16:37:00.9247917Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9248096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9248161Z res = mod(**inputs) 2025-11-03T16:37:00.9248416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9248478Z outputs = self.mobilebert( 2025-11-03T16:37:00.9248742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9248808Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9249071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9249154Z layer_outputs = layer_module( 2025-11-03T16:37:00.9249419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9249506Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9249769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9249876Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9250143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9250256Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9250273Z 2025-11-03T16:37:00.9250366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9250547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9250611Z res = mod(**inputs) 2025-11-03T16:37:00.9250867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9250938Z outputs = self.mobilebert( 2025-11-03T16:37:00.9251194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9251268Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9251526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9251591Z layer_outputs = layer_module( 2025-11-03T16:37:00.9251873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9251960Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9252229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9252344Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9252602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9252686Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9252689Z 2025-11-03T16:37:00.9252783Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9252973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9253033Z res = mod(**inputs) 2025-11-03T16:37:00.9253301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9253385Z outputs = self.mobilebert( 2025-11-03T16:37:00.9253651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9253724Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9253983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9254054Z layer_outputs = layer_module( 2025-11-03T16:37:00.9254313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9254399Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9254669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9254781Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9255060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9255188Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9255451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9255534Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9255538Z 2025-11-03T16:37:00.9255630Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9255818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9255876Z res = mod(**inputs) 2025-11-03T16:37:00.9256156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9256224Z outputs = self.mobilebert( 2025-11-03T16:37:00.9256478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9256550Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9256809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9256881Z layer_outputs = layer_module( 2025-11-03T16:37:00.9257135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9257227Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9257529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9257634Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9257907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9257985Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9257988Z 2025-11-03T16:37:00.9258090Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9258271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9258335Z res = mod(**inputs) 2025-11-03T16:37:00.9258603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9258666Z outputs = self.mobilebert( 2025-11-03T16:37:00.9258932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9258998Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9259258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9259350Z layer_outputs = layer_module( 2025-11-03T16:37:00.9259614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9259705Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9259965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9260074Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9260337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9260449Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9260453Z 2025-11-03T16:37:00.9260547Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9260749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9260814Z res = mod(**inputs) 2025-11-03T16:37:00.9261071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9261141Z outputs = self.mobilebert( 2025-11-03T16:37:00.9261400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9261465Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9261733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9261799Z layer_outputs = layer_module( 2025-11-03T16:37:00.9262077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9262166Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9262430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9262543Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9262801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9262886Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9262889Z 2025-11-03T16:37:00.9262982Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9263170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9263242Z res = mod(**inputs) 2025-11-03T16:37:00.9263500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9263574Z outputs = self.mobilebert( 2025-11-03T16:37:00.9263833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9263905Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9264161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9264232Z layer_outputs = layer_module( 2025-11-03T16:37:00.9264490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9264574Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9264843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9264957Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9265244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9265355Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9265610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9265702Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9265705Z 2025-11-03T16:37:00.9265797Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9265985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9266045Z res = mod(**inputs) 2025-11-03T16:37:00.9266309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9266395Z outputs = self.mobilebert( 2025-11-03T16:37:00.9266654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9266728Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9266996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9267069Z layer_outputs = layer_module( 2025-11-03T16:37:00.9267333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9267448Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9267735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9267814Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9267820Z 2025-11-03T16:37:00.9267923Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9268112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9268181Z res = mod(**inputs) 2025-11-03T16:37:00.9268452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9268518Z outputs = self.mobilebert( 2025-11-03T16:37:00.9268791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9268859Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9269146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9269215Z layer_outputs = layer_module( 2025-11-03T16:37:00.9269481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9269604Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9269870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9269983Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9269987Z 2025-11-03T16:37:00.9270081Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9270275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9270336Z res = mod(**inputs) 2025-11-03T16:37:00.9270603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9270678Z outputs = self.mobilebert( 2025-11-03T16:37:00.9270945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9271035Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9271303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9271369Z layer_outputs = layer_module( 2025-11-03T16:37:00.9271639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9271789Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9272061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9272150Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9272153Z 2025-11-03T16:37:00.9272273Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9272464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9272524Z res = mod(**inputs) 2025-11-03T16:37:00.9272798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9272862Z outputs = self.mobilebert( 2025-11-03T16:37:00.9273133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9273200Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9273471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9273562Z layer_outputs = layer_module( 2025-11-03T16:37:00.9273832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9273996Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9274335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9274464Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9274737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9274824Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9274837Z 2025-11-03T16:37:00.9274936Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9275151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9275223Z res = mod(**inputs) 2025-11-03T16:37:00.9275509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9275587Z outputs = self.mobilebert( 2025-11-03T16:37:00.9275858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9275926Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9276211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9276277Z layer_outputs = layer_module( 2025-11-03T16:37:00.9276570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9276725Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9277002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9277153Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9277431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9277522Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9277525Z 2025-11-03T16:37:00.9277625Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9277821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9277882Z res = mod(**inputs) 2025-11-03T16:37:00.9278158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9278235Z outputs = self.mobilebert( 2025-11-03T16:37:00.9278510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9278604Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9278875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9278942Z layer_outputs = layer_module( 2025-11-03T16:37:00.9279228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9279376Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9279651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9279771Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9280065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9280186Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9280459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9280554Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9280558Z 2025-11-03T16:37:00.9280657Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9280854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9280914Z res = mod(**inputs) 2025-11-03T16:37:00.9281195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9281288Z outputs = self.mobilebert( 2025-11-03T16:37:00.9281560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9281638Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9281908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9281981Z layer_outputs = layer_module( 2025-11-03T16:37:00.9282252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9282409Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9282689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9282798Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9283078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9283178Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9283183Z 2025-11-03T16:37:00.9283290Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9283498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9283560Z res = mod(**inputs) 2025-11-03T16:37:00.9283833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9283898Z outputs = self.mobilebert( 2025-11-03T16:37:00.9284174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9284242Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9284574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9284674Z layer_outputs = layer_module( 2025-11-03T16:37:00.9284939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9285028Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9285292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9285366Z self_outputs = self.self( 2025-11-03T16:37:00.9285640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9285707Z self.value(value_tensor) 2025-11-03T16:37:00.9285710Z 2025-11-03T16:37:00.9285813Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9286018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9286091Z res = mod(**inputs) 2025-11-03T16:37:00.9286356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9286421Z outputs = self.mobilebert( 2025-11-03T16:37:00.9286703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9286770Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9287049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9287115Z layer_outputs = layer_module( 2025-11-03T16:37:00.9287408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9287560Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9287827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9287940Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9288208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9288294Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9288298Z 2025-11-03T16:37:00.9288395Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9288579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9288647Z res = mod(**inputs) 2025-11-03T16:37:00.9288914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9288987Z outputs = self.mobilebert( 2025-11-03T16:37:00.9289252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9289345Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9289618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9289683Z layer_outputs = layer_module( 2025-11-03T16:37:00.9289944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9290089Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9290354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9290455Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9290712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9290817Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9291076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9291166Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9291170Z 2025-11-03T16:37:00.9291263Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9291450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9291509Z res = mod(**inputs) 2025-11-03T16:37:00.9291766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9291852Z outputs = self.mobilebert( 2025-11-03T16:37:00.9292110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9292186Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9292443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9292507Z layer_outputs = layer_module( 2025-11-03T16:37:00.9292771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9292849Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9293113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9293177Z self_outputs = self.self( 2025-11-03T16:37:00.9293456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9293525Z self.query(query_tensor) 2025-11-03T16:37:00.9293529Z 2025-11-03T16:37:00.9293620Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9293809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9293868Z res = mod(**inputs) 2025-11-03T16:37:00.9294131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9294194Z outputs = self.mobilebert( 2025-11-03T16:37:00.9294452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9294526Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9294785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9294857Z layer_outputs = layer_module( 2025-11-03T16:37:00.9295129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9295213Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9295470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9295535Z self_outputs = self.self( 2025-11-03T16:37:00.9295799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9295861Z self.key(key_tensor) 2025-11-03T16:37:00.9295864Z 2025-11-03T16:37:00.9295946Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9296020Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9296115Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9296305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9296380Z res = mod(**inputs) 2025-11-03T16:37:00.9296648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9296711Z outputs = self.mobilebert( 2025-11-03T16:37:00.9296979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9297054Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9297309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9297380Z layer_outputs = layer_module( 2025-11-03T16:37:00.9297656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9297744Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9298002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9298115Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9298380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9298458Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9298461Z 2025-11-03T16:37:00.9298562Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9298741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9298800Z res = mod(**inputs) 2025-11-03T16:37:00.9299081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9299148Z outputs = self.mobilebert( 2025-11-03T16:37:00.9299417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9299482Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9299748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9299813Z layer_outputs = layer_module( 2025-11-03T16:37:00.9300071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9300156Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9300414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9300535Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9300794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9300932Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9301199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9301284Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9301288Z 2025-11-03T16:37:00.9301387Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9301570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9301635Z res = mod(**inputs) 2025-11-03T16:37:00.9301906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9301971Z outputs = self.mobilebert( 2025-11-03T16:37:00.9302237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9302324Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9302598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9302663Z layer_outputs = layer_module( 2025-11-03T16:37:00.9302923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9303019Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9303279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9303413Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9303672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9303755Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9303758Z 2025-11-03T16:37:00.9303849Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9304029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9304092Z res = mod(**inputs) 2025-11-03T16:37:00.9304347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9304417Z outputs = self.mobilebert( 2025-11-03T16:37:00.9304677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9304759Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9305024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9305092Z layer_outputs = layer_module( 2025-11-03T16:37:00.9305358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9305444Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9305707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9305808Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9306066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9306177Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9306182Z 2025-11-03T16:37:00.9306277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9306466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9306540Z res = mod(**inputs) 2025-11-03T16:37:00.9306806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9306870Z outputs = self.mobilebert( 2025-11-03T16:37:00.9307131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9307205Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9307462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9307533Z layer_outputs = layer_module( 2025-11-03T16:37:00.9307793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9307879Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9308164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9308279Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9308541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9308616Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9308620Z 2025-11-03T16:37:00.9308718Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9308897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9308954Z res = mod(**inputs) 2025-11-03T16:37:00.9309231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9309298Z outputs = self.mobilebert( 2025-11-03T16:37:00.9309561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9309627Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9309884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9309955Z layer_outputs = layer_module( 2025-11-03T16:37:00.9310212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9310305Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9310576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9310699Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9310961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9311073Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9311341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9311426Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9311429Z 2025-11-03T16:37:00.9311529Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9311714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9311770Z res = mod(**inputs) 2025-11-03T16:37:00.9312040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9312105Z outputs = self.mobilebert( 2025-11-03T16:37:00.9312370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9312452Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9312723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9312789Z layer_outputs = layer_module( 2025-11-03T16:37:00.9313051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9313147Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9313560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9313676Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9313942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9314101Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9314114Z 2025-11-03T16:37:00.9314214Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9314402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9314474Z res = mod(**inputs) 2025-11-03T16:37:00.9314740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9314819Z outputs = self.mobilebert( 2025-11-03T16:37:00.9315091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9315189Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9315459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9315528Z layer_outputs = layer_module( 2025-11-03T16:37:00.9315791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9315875Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9316133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9316241Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9316500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9316635Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9316640Z 2025-11-03T16:37:00.9316735Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9316925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9316984Z res = mod(**inputs) 2025-11-03T16:37:00.9317243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9317317Z outputs = self.mobilebert( 2025-11-03T16:37:00.9317579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9317651Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9317918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9317983Z layer_outputs = layer_module( 2025-11-03T16:37:00.9318250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9318337Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9318627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9318742Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9319022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9319099Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9319101Z 2025-11-03T16:37:00.9319194Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9319382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9319440Z res = mod(**inputs) 2025-11-03T16:37:00.9319709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9319790Z outputs = self.mobilebert( 2025-11-03T16:37:00.9320049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9320121Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9320379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9320453Z layer_outputs = layer_module( 2025-11-03T16:37:00.9320710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9320804Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9321082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9321200Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9321468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9321580Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9321845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9321928Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9321931Z 2025-11-03T16:37:00.9322028Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9322210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9322270Z res = mod(**inputs) 2025-11-03T16:37:00.9322548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9322615Z outputs = self.mobilebert( 2025-11-03T16:37:00.9322877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9322943Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9323198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9323271Z layer_outputs = layer_module( 2025-11-03T16:37:00.9323525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9323615Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9323874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9323981Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9324250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9324345Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9324348Z 2025-11-03T16:37:00.9324446Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9324625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9324691Z res = mod(**inputs) 2025-11-03T16:37:00.9324944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9325006Z outputs = self.mobilebert( 2025-11-03T16:37:00.9325268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9325335Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9325596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9325687Z layer_outputs = layer_module( 2025-11-03T16:37:00.9325950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9326035Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9326292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9326399Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9326654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9326779Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9326783Z 2025-11-03T16:37:00.9326877Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9327061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9327125Z res = mod(**inputs) 2025-11-03T16:37:00.9327382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9327454Z outputs = self.mobilebert( 2025-11-03T16:37:00.9327710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9327783Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9328040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9328119Z layer_outputs = layer_module( 2025-11-03T16:37:00.9328388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9328475Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9328739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9328854Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9329111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9329196Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9329199Z 2025-11-03T16:37:00.9329293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9329482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9329542Z res = mod(**inputs) 2025-11-03T16:37:00.9329810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9329895Z outputs = self.mobilebert( 2025-11-03T16:37:00.9330150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9330224Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9330481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9330554Z layer_outputs = layer_module( 2025-11-03T16:37:00.9330812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9330897Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9331163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9331277Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9331563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9331676Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9331942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9332025Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9332028Z 2025-11-03T16:37:00.9332122Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9332312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9332370Z res = mod(**inputs) 2025-11-03T16:37:00.9332652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9332719Z outputs = self.mobilebert( 2025-11-03T16:37:00.9332975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9333048Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9333303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9333375Z layer_outputs = layer_module( 2025-11-03T16:37:00.9333635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9333753Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9334024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9334103Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9334109Z 2025-11-03T16:37:00.9334212Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9334394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9334458Z res = mod(**inputs) 2025-11-03T16:37:00.9334714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9334780Z outputs = self.mobilebert( 2025-11-03T16:37:00.9335046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9335111Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9335376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9335439Z layer_outputs = layer_module( 2025-11-03T16:37:00.9335701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9335828Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9336084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9336194Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9336197Z 2025-11-03T16:37:00.9336290Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9336477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9336534Z res = mod(**inputs) 2025-11-03T16:37:00.9336801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9336864Z outputs = self.mobilebert( 2025-11-03T16:37:00.9337120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9337219Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9337476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9337548Z layer_outputs = layer_module( 2025-11-03T16:37:00.9337804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9337949Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9338213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9338315Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9338320Z 2025-11-03T16:37:00.9338421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9338605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9338670Z res = mod(**inputs) 2025-11-03T16:37:00.9338928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9338991Z outputs = self.mobilebert( 2025-11-03T16:37:00.9339255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9339322Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9339584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9339661Z layer_outputs = layer_module( 2025-11-03T16:37:00.9339919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9340071Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9340327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9340441Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9340696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9340788Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9340792Z 2025-11-03T16:37:00.9340885Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9341065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9341131Z res = mod(**inputs) 2025-11-03T16:37:00.9341386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9341477Z outputs = self.mobilebert( 2025-11-03T16:37:00.9341738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9341803Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9342070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9342134Z layer_outputs = layer_module( 2025-11-03T16:37:00.9342401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9342548Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9342816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9342945Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9343205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9343292Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9343295Z 2025-11-03T16:37:00.9343389Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9343579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9343637Z res = mod(**inputs) 2025-11-03T16:37:00.9343895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9343986Z outputs = self.mobilebert( 2025-11-03T16:37:00.9344244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9344321Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9344578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9344652Z layer_outputs = layer_module( 2025-11-03T16:37:00.9344907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9345048Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9345312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9345441Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9345711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9345827Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9346098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9346183Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9346186Z 2025-11-03T16:37:00.9346280Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9346472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9346530Z res = mod(**inputs) 2025-11-03T16:37:00.9346798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9346870Z outputs = self.mobilebert( 2025-11-03T16:37:00.9347132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9347231Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9347490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9347561Z layer_outputs = layer_module( 2025-11-03T16:37:00.9347819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9347974Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9348233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9348334Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9348597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9348690Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9348693Z 2025-11-03T16:37:00.9348793Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9348976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9349034Z res = mod(**inputs) 2025-11-03T16:37:00.9349298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9349361Z outputs = self.mobilebert( 2025-11-03T16:37:00.9349623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9349690Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9349995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9350063Z layer_outputs = layer_module( 2025-11-03T16:37:00.9350322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9350409Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9350664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9350737Z self_outputs = self.self( 2025-11-03T16:37:00.9351002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9351068Z self.value(value_tensor) 2025-11-03T16:37:00.9351079Z 2025-11-03T16:37:00.9351175Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9351380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9351447Z res = mod(**inputs) 2025-11-03T16:37:00.9351719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9351791Z outputs = self.mobilebert( 2025-11-03T16:37:00.9352063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9352130Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9352408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9352474Z layer_outputs = layer_module( 2025-11-03T16:37:00.9352751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9352910Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9353188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9353319Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9353589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9353673Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9353677Z 2025-11-03T16:37:00.9353774Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9353970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9354091Z res = mod(**inputs) 2025-11-03T16:37:00.9354374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9354447Z outputs = self.mobilebert( 2025-11-03T16:37:00.9354720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9354819Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9355098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9355163Z layer_outputs = layer_module( 2025-11-03T16:37:00.9355436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9355586Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9355866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9355986Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9356260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9356346Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9356612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9356706Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9356709Z 2025-11-03T16:37:00.9356806Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9356999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9357061Z res = mod(**inputs) 2025-11-03T16:37:00.9357355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9357425Z outputs = self.mobilebert( 2025-11-03T16:37:00.9357693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9357772Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9358039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9358112Z layer_outputs = layer_module( 2025-11-03T16:37:00.9358376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9358457Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9358734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9358803Z self_outputs = self.self( 2025-11-03T16:37:00.9359085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9359154Z self.query(query_tensor) 2025-11-03T16:37:00.9359173Z 2025-11-03T16:37:00.9359275Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9359462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9359522Z res = mod(**inputs) 2025-11-03T16:37:00.9359793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9359857Z outputs = self.mobilebert( 2025-11-03T16:37:00.9360133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9360201Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9360468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9360543Z layer_outputs = layer_module( 2025-11-03T16:37:00.9360827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9360916Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9361183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9361249Z self_outputs = self.self( 2025-11-03T16:37:00.9361522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9361593Z self.key(key_tensor) 2025-11-03T16:37:00.9361596Z 2025-11-03T16:37:00.9361681Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9361758Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9361877Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9362064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9362128Z res = mod(**inputs) 2025-11-03T16:37:00.9362398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9362462Z outputs = self.mobilebert( 2025-11-03T16:37:00.9362730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9362795Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9363059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9363131Z layer_outputs = layer_module( 2025-11-03T16:37:00.9363409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9363495Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9363763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9363876Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9364146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9364226Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9364230Z 2025-11-03T16:37:00.9364334Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9364521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9364588Z res = mod(**inputs) 2025-11-03T16:37:00.9364856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9364922Z outputs = self.mobilebert( 2025-11-03T16:37:00.9365212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9365280Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9365550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9365618Z layer_outputs = layer_module( 2025-11-03T16:37:00.9365881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9365966Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9366232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9366357Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9366621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9366766Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9367041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9367125Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9367129Z 2025-11-03T16:37:00.9367229Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9367409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9367473Z res = mod(**inputs) 2025-11-03T16:37:00.9367734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9367818Z outputs = self.mobilebert( 2025-11-03T16:37:00.9368074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9368141Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9368405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9368467Z layer_outputs = layer_module( 2025-11-03T16:37:00.9368727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9368814Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9369069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9369205Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9369467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9369554Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9369557Z 2025-11-03T16:37:00.9369652Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9369842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9369899Z res = mod(**inputs) 2025-11-03T16:37:00.9370153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9370225Z outputs = self.mobilebert( 2025-11-03T16:37:00.9370481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9370554Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9370812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9370893Z layer_outputs = layer_module( 2025-11-03T16:37:00.9371165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9371253Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9371520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9371624Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9371891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9371994Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9371999Z 2025-11-03T16:37:00.9372093Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9372282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9372358Z res = mod(**inputs) 2025-11-03T16:37:00.9372622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9372688Z outputs = self.mobilebert( 2025-11-03T16:37:00.9372948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9373022Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9373278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9373352Z layer_outputs = layer_module( 2025-11-03T16:37:00.9373625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9373720Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9373979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9374095Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9374366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9374444Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9374447Z 2025-11-03T16:37:00.9374548Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9374730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9374788Z res = mod(**inputs) 2025-11-03T16:37:00.9375066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9375135Z outputs = self.mobilebert( 2025-11-03T16:37:00.9375397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9375461Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9375732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9375797Z layer_outputs = layer_module( 2025-11-03T16:37:00.9376050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9376145Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9376402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9376523Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9376779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9376908Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9377172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9377257Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9377260Z 2025-11-03T16:37:00.9377362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9377540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9377606Z res = mod(**inputs) 2025-11-03T16:37:00.9377867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9377932Z outputs = self.mobilebert( 2025-11-03T16:37:00.9378216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9378283Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9378545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9378610Z layer_outputs = layer_module( 2025-11-03T16:37:00.9378865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9378959Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9379215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9379340Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9379602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9379688Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9379692Z 2025-11-03T16:37:00.9379785Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9379965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9380032Z res = mod(**inputs) 2025-11-03T16:37:00.9380287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9380356Z outputs = self.mobilebert( 2025-11-03T16:37:00.9380611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9380691Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9380956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9381021Z layer_outputs = layer_module( 2025-11-03T16:37:00.9381284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9381370Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9381634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9381734Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9381991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9382103Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9382106Z 2025-11-03T16:37:00.9382202Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9382401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9382478Z res = mod(**inputs) 2025-11-03T16:37:00.9382759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9382830Z outputs = self.mobilebert( 2025-11-03T16:37:00.9383104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9383182Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9383454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9383529Z layer_outputs = layer_module( 2025-11-03T16:37:00.9383812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9383919Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9384191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9384312Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9384588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9384667Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9384671Z 2025-11-03T16:37:00.9384774Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9384963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9385024Z res = mod(**inputs) 2025-11-03T16:37:00.9385319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9385389Z outputs = self.mobilebert( 2025-11-03T16:37:00.9385671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9385741Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9386020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9386095Z layer_outputs = layer_module( 2025-11-03T16:37:00.9386372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9386470Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9386768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9386896Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9387177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9387295Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9387580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9387669Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9387672Z 2025-11-03T16:37:00.9387778Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9387975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9388035Z res = mod(**inputs) 2025-11-03T16:37:00.9388324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9388394Z outputs = self.mobilebert( 2025-11-03T16:37:00.9388691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9388761Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9389040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9389109Z layer_outputs = layer_module( 2025-11-03T16:37:00.9389384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9389482Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9389757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9389868Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9390141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9390239Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9390249Z 2025-11-03T16:37:00.9390347Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9390539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9390609Z res = mod(**inputs) 2025-11-03T16:37:00.9390880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9390954Z outputs = self.mobilebert( 2025-11-03T16:37:00.9391226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9391321Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9391606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9391677Z layer_outputs = layer_module( 2025-11-03T16:37:00.9391954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9392042Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9392319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9392433Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9392711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9392843Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9392847Z 2025-11-03T16:37:00.9392947Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9393146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9393208Z res = mod(**inputs) 2025-11-03T16:37:00.9393484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9393556Z outputs = self.mobilebert( 2025-11-03T16:37:00.9393834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9393909Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9394274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9394354Z layer_outputs = layer_module( 2025-11-03T16:37:00.9394648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9394758Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9395031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9395147Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9395429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9395507Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9395511Z 2025-11-03T16:37:00.9395605Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9395797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9395857Z res = mod(**inputs) 2025-11-03T16:37:00.9396124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9396205Z outputs = self.mobilebert( 2025-11-03T16:37:00.9396462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9396536Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9396793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9396866Z layer_outputs = layer_module( 2025-11-03T16:37:00.9397125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9397217Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9397488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9397603Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9397869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9397980Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9398243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9398325Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9398328Z 2025-11-03T16:37:00.9398426Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9398602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9398659Z res = mod(**inputs) 2025-11-03T16:37:00.9398938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9399005Z outputs = self.mobilebert( 2025-11-03T16:37:00.9399270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9399334Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9399590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9399666Z layer_outputs = layer_module( 2025-11-03T16:37:00.9399918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9400034Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9400292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9400376Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9400381Z 2025-11-03T16:37:00.9400490Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9400676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9400743Z res = mod(**inputs) 2025-11-03T16:37:00.9401006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9401076Z outputs = self.mobilebert( 2025-11-03T16:37:00.9401340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9401405Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9401679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9401746Z layer_outputs = layer_module( 2025-11-03T16:37:00.9402018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9402148Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9402413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9402515Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9402518Z 2025-11-03T16:37:00.9402612Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9402800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9402858Z res = mod(**inputs) 2025-11-03T16:37:00.9403137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9403203Z outputs = self.mobilebert( 2025-11-03T16:37:00.9403472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9403547Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9403809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9403881Z layer_outputs = layer_module( 2025-11-03T16:37:00.9404138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9404289Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9404560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9404650Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9404653Z 2025-11-03T16:37:00.9404755Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9404936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9405005Z res = mod(**inputs) 2025-11-03T16:37:00.9405263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9405328Z outputs = self.mobilebert( 2025-11-03T16:37:00.9405592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9405656Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9405925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9405990Z layer_outputs = layer_module( 2025-11-03T16:37:00.9406254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9406417Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9406678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9406801Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9407063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9407154Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9407157Z 2025-11-03T16:37:00.9407251Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9407442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9407501Z res = mod(**inputs) 2025-11-03T16:37:00.9407762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9407853Z outputs = self.mobilebert( 2025-11-03T16:37:00.9408112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9408186Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9408445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9408509Z layer_outputs = layer_module( 2025-11-03T16:37:00.9408776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9408923Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9409207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9409325Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9409592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9409670Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9409673Z 2025-11-03T16:37:00.9409768Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9409957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9410015Z res = mod(**inputs) 2025-11-03T16:37:00.9410280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9410360Z outputs = self.mobilebert( 2025-11-03T16:37:00.9410620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9410695Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9410953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9411024Z layer_outputs = layer_module( 2025-11-03T16:37:00.9411280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9411429Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9411688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9411803Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9412065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9412195Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9412461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9412543Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9412546Z 2025-11-03T16:37:00.9412648Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9412830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9412888Z res = mod(**inputs) 2025-11-03T16:37:00.9413151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9413359Z outputs = self.mobilebert( 2025-11-03T16:37:00.9413640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9413756Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9414014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9414086Z layer_outputs = layer_module( 2025-11-03T16:37:00.9414344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9414500Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9414761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9414864Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9415152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9415234Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9415238Z 2025-11-03T16:37:00.9415339Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9415521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9415587Z res = mod(**inputs) 2025-11-03T16:37:00.9415844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9415909Z outputs = self.mobilebert( 2025-11-03T16:37:00.9416174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9416241Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9416533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9416599Z layer_outputs = layer_module( 2025-11-03T16:37:00.9416856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9416941Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9417195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9417271Z self_outputs = self.self( 2025-11-03T16:37:00.9417526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9417599Z self.value(value_tensor) 2025-11-03T16:37:00.9417602Z 2025-11-03T16:37:00.9417697Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9417880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9417947Z res = mod(**inputs) 2025-11-03T16:37:00.9418233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9418305Z outputs = self.mobilebert( 2025-11-03T16:37:00.9418561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9418628Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9418893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9418958Z layer_outputs = layer_module( 2025-11-03T16:37:00.9419224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9419373Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9419637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9419759Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9420018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9420102Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9420106Z 2025-11-03T16:37:00.9420198Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9420384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9420445Z res = mod(**inputs) 2025-11-03T16:37:00.9420711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9420791Z outputs = self.mobilebert( 2025-11-03T16:37:00.9421053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9421131Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9421389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9421469Z layer_outputs = layer_module( 2025-11-03T16:37:00.9421727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9421873Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9422156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9422258Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9422523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9422606Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9422873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9422958Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9422961Z 2025-11-03T16:37:00.9423053Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9423243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9423302Z res = mod(**inputs) 2025-11-03T16:37:00.9423568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9423632Z outputs = self.mobilebert( 2025-11-03T16:37:00.9423891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9423984Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9424243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9424315Z layer_outputs = layer_module( 2025-11-03T16:37:00.9424572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9424658Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9424913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9424978Z self_outputs = self.self( 2025-11-03T16:37:00.9425245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9425330Z self.query(query_tensor) 2025-11-03T16:37:00.9425334Z 2025-11-03T16:37:00.9425436Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9425621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9425681Z res = mod(**inputs) 2025-11-03T16:37:00.9425951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9426016Z outputs = self.mobilebert( 2025-11-03T16:37:00.9426288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9426354Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9426643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9426709Z layer_outputs = layer_module( 2025-11-03T16:37:00.9426968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9427053Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9427311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9427383Z self_outputs = self.self( 2025-11-03T16:37:00.9427645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9427707Z self.key(key_tensor) 2025-11-03T16:37:00.9427710Z 2025-11-03T16:37:00.9427791Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9427865Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9427985Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9428170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9428232Z res = mod(**inputs) 2025-11-03T16:37:00.9428496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9428559Z outputs = self.mobilebert( 2025-11-03T16:37:00.9428824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9428891Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9429167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9429231Z layer_outputs = layer_module( 2025-11-03T16:37:00.9429500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9429587Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9429869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9429991Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9430256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9430335Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9430346Z 2025-11-03T16:37:00.9430442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9430631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9430697Z res = mod(**inputs) 2025-11-03T16:37:00.9430965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9431038Z outputs = self.mobilebert( 2025-11-03T16:37:00.9431319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9431388Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9431666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9431733Z layer_outputs = layer_module( 2025-11-03T16:37:00.9432004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9432083Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9432348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9432488Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9432755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9432884Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9433151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9433245Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9433248Z 2025-11-03T16:37:00.9433342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9433529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9433597Z res = mod(**inputs) 2025-11-03T16:37:00.9433880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9433956Z outputs = self.mobilebert( 2025-11-03T16:37:00.9434278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9434356Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9434631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9434697Z layer_outputs = layer_module( 2025-11-03T16:37:00.9434970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9435061Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9435340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9435449Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9435721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9435844Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9435848Z 2025-11-03T16:37:00.9435947Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9436147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9436214Z res = mod(**inputs) 2025-11-03T16:37:00.9436484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9436564Z outputs = self.mobilebert( 2025-11-03T16:37:00.9436835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9436917Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9437187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9437281Z layer_outputs = layer_module( 2025-11-03T16:37:00.9437548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9437638Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9437915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9438018Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9438290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9438395Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9438400Z 2025-11-03T16:37:00.9438520Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9438707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9438768Z res = mod(**inputs) 2025-11-03T16:37:00.9439041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9439107Z outputs = self.mobilebert( 2025-11-03T16:37:00.9439376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9439442Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9439706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9439782Z layer_outputs = layer_module( 2025-11-03T16:37:00.9440064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9440162Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9440433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9440553Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9440830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9440910Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9440913Z 2025-11-03T16:37:00.9441016Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9441204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9441270Z res = mod(**inputs) 2025-11-03T16:37:00.9441539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9441606Z outputs = self.mobilebert( 2025-11-03T16:37:00.9441893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9441960Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9442232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9442298Z layer_outputs = layer_module( 2025-11-03T16:37:00.9442563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9442659Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9442924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9443052Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9443367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9443491Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9443756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9443842Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9443845Z 2025-11-03T16:37:00.9443948Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9444134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9444199Z res = mod(**inputs) 2025-11-03T16:37:00.9444478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9444552Z outputs = self.mobilebert( 2025-11-03T16:37:00.9444818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9444888Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9445154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9445221Z layer_outputs = layer_module( 2025-11-03T16:37:00.9445495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9445580Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9445835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9445959Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9446215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9446302Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9446305Z 2025-11-03T16:37:00.9446398Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9446585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9446644Z res = mod(**inputs) 2025-11-03T16:37:00.9446899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9446972Z outputs = self.mobilebert( 2025-11-03T16:37:00.9447227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9447304Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9447560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9447640Z layer_outputs = layer_module( 2025-11-03T16:37:00.9447908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9447994Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9448264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9448366Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9448629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9448734Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9448737Z 2025-11-03T16:37:00.9448831Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9449024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9449100Z res = mod(**inputs) 2025-11-03T16:37:00.9449361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9449425Z outputs = self.mobilebert( 2025-11-03T16:37:00.9449681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9449756Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9450016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9450087Z layer_outputs = layer_module( 2025-11-03T16:37:00.9450362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9450459Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9450718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9450832Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9451097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9451173Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9451176Z 2025-11-03T16:37:00.9451276Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9451457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9451515Z res = mod(**inputs) 2025-11-03T16:37:00.9451808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9451876Z outputs = self.mobilebert( 2025-11-03T16:37:00.9452141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9452207Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9452469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9452532Z layer_outputs = layer_module( 2025-11-03T16:37:00.9452789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9452883Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9453146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9453267Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9453541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9453652Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9453930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9454014Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9454017Z 2025-11-03T16:37:00.9454116Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9454297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9454363Z res = mod(**inputs) 2025-11-03T16:37:00.9454622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9454687Z outputs = self.mobilebert( 2025-11-03T16:37:00.9454975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9455042Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9455308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9455371Z layer_outputs = layer_module( 2025-11-03T16:37:00.9455629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9455722Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9455984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9456108Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9456371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9456457Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9456460Z 2025-11-03T16:37:00.9456555Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9456738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9456804Z res = mod(**inputs) 2025-11-03T16:37:00.9457061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9457133Z outputs = self.mobilebert( 2025-11-03T16:37:00.9457415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9457484Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9457753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9457820Z layer_outputs = layer_module( 2025-11-03T16:37:00.9458086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9458171Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9458434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9458534Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9458790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9458901Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9458905Z 2025-11-03T16:37:00.9458998Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9459203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9459261Z res = mod(**inputs) 2025-11-03T16:37:00.9459527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9459591Z outputs = self.mobilebert( 2025-11-03T16:37:00.9459849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9459922Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9460179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9460251Z layer_outputs = layer_module( 2025-11-03T16:37:00.9460514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9460618Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9460888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9461001Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9461279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9461356Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9461359Z 2025-11-03T16:37:00.9461462Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9461643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9461703Z res = mod(**inputs) 2025-11-03T16:37:00.9461986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9462054Z outputs = self.mobilebert( 2025-11-03T16:37:00.9462319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9462384Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9462641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9462712Z layer_outputs = layer_module( 2025-11-03T16:37:00.9462968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9463061Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9463331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9463455Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9463715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9463829Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9464092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9464175Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9464178Z 2025-11-03T16:37:00.9464277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9464460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9464518Z res = mod(**inputs) 2025-11-03T16:37:00.9464784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9464851Z outputs = self.mobilebert( 2025-11-03T16:37:00.9465134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9465199Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9465468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9465532Z layer_outputs = layer_module( 2025-11-03T16:37:00.9465790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9465912Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9466176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9466260Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9466281Z 2025-11-03T16:37:00.9466380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9466565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9466632Z res = mod(**inputs) 2025-11-03T16:37:00.9466890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9466962Z outputs = self.mobilebert( 2025-11-03T16:37:00.9467219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9467292Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9467549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9467629Z layer_outputs = layer_module( 2025-11-03T16:37:00.9467892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9468004Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9468268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9468369Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9468373Z 2025-11-03T16:37:00.9468476Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9468658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9468715Z res = mod(**inputs) 2025-11-03T16:37:00.9468999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9469067Z outputs = self.mobilebert( 2025-11-03T16:37:00.9469331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9469401Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9469657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9469732Z layer_outputs = layer_module( 2025-11-03T16:37:00.9469989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9470143Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9470403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9470493Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9470504Z 2025-11-03T16:37:00.9470597Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9470796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9470862Z res = mod(**inputs) 2025-11-03T16:37:00.9471121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9471192Z outputs = self.mobilebert( 2025-11-03T16:37:00.9471451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9471516Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9471781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9471850Z layer_outputs = layer_module( 2025-11-03T16:37:00.9472113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9472276Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9472540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9472652Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9472911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9473004Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9473007Z 2025-11-03T16:37:00.9473100Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9473288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9473359Z res = mod(**inputs) 2025-11-03T16:37:00.9473618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9473693Z outputs = self.mobilebert( 2025-11-03T16:37:00.9473965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9474104Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9474387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9474463Z layer_outputs = layer_module( 2025-11-03T16:37:00.9474734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9474907Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9475217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9475338Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9475615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9475697Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9475700Z 2025-11-03T16:37:00.9475799Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9476002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9476064Z res = mod(**inputs) 2025-11-03T16:37:00.9476347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9476415Z outputs = self.mobilebert( 2025-11-03T16:37:00.9476700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9476786Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9477067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9477143Z layer_outputs = layer_module( 2025-11-03T16:37:00.9477421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9477579Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9477859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9477979Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9478266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9478403Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9478684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9478773Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9478776Z 2025-11-03T16:37:00.9478881Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9479076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9479137Z res = mod(**inputs) 2025-11-03T16:37:00.9479415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9479484Z outputs = self.mobilebert( 2025-11-03T16:37:00.9479789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9479863Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9480137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9480211Z layer_outputs = layer_module( 2025-11-03T16:37:00.9480484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9480647Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9480922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9481049Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9481327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9481411Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9481429Z 2025-11-03T16:37:00.9481528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9481726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9481795Z res = mod(**inputs) 2025-11-03T16:37:00.9482068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9482145Z outputs = self.mobilebert( 2025-11-03T16:37:00.9482420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9482488Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9482785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9482869Z layer_outputs = layer_module( 2025-11-03T16:37:00.9483174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9483258Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9483531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9483608Z self_outputs = self.self( 2025-11-03T16:37:00.9483877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9483953Z self.value(value_tensor) 2025-11-03T16:37:00.9483956Z 2025-11-03T16:37:00.9484056Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9484261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9484323Z res = mod(**inputs) 2025-11-03T16:37:00.9484616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9484692Z outputs = self.mobilebert( 2025-11-03T16:37:00.9484968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9485044Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9485315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9485382Z layer_outputs = layer_module( 2025-11-03T16:37:00.9485663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9485835Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9486119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9486228Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9486507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9486586Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9486590Z 2025-11-03T16:37:00.9486686Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9486881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9486941Z res = mod(**inputs) 2025-11-03T16:37:00.9487233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9487305Z outputs = self.mobilebert( 2025-11-03T16:37:00.9487577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9487657Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9487925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9487997Z layer_outputs = layer_module( 2025-11-03T16:37:00.9488267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9488427Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9488700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9488806Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9489087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9489181Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9501866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9502021Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9502028Z 2025-11-03T16:37:00.9502152Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9502362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9502429Z res = mod(**inputs) 2025-11-03T16:37:00.9502746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9502823Z outputs = self.mobilebert( 2025-11-03T16:37:00.9503100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9503262Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9503533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9503606Z layer_outputs = layer_module( 2025-11-03T16:37:00.9503864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9503958Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9504214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9504293Z self_outputs = self.self( 2025-11-03T16:37:00.9504588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9504659Z self.query(query_tensor) 2025-11-03T16:37:00.9504664Z 2025-11-03T16:37:00.9504777Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9504973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9505041Z res = mod(**inputs) 2025-11-03T16:37:00.9505308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9505385Z outputs = self.mobilebert( 2025-11-03T16:37:00.9505653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9505724Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9506028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9506100Z layer_outputs = layer_module( 2025-11-03T16:37:00.9506376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9506461Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9506723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9506800Z self_outputs = self.self( 2025-11-03T16:37:00.9507062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9507135Z self.key(key_tensor) 2025-11-03T16:37:00.9507139Z 2025-11-03T16:37:00.9507222Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9507300Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9507411Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9507603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9507701Z res = mod(**inputs) 2025-11-03T16:37:00.9507961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9508036Z outputs = self.mobilebert( 2025-11-03T16:37:00.9508296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9508364Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9508631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9508696Z layer_outputs = layer_module( 2025-11-03T16:37:00.9508959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9509039Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9509315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9509440Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9509702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9509788Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9509791Z 2025-11-03T16:37:00.9509888Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9510082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9510142Z res = mod(**inputs) 2025-11-03T16:37:00.9510420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9510494Z outputs = self.mobilebert( 2025-11-03T16:37:00.9510756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9510833Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9511095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9511159Z layer_outputs = layer_module( 2025-11-03T16:37:00.9511426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9511504Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9511775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9511905Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9512172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9512297Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9512554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9512649Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9512653Z 2025-11-03T16:37:00.9512746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9512939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9512999Z res = mod(**inputs) 2025-11-03T16:37:00.9513501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9513581Z outputs = self.mobilebert( 2025-11-03T16:37:00.9513841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9513977Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9514296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9514375Z layer_outputs = layer_module( 2025-11-03T16:37:00.9514643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9514737Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9515018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9515142Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9515408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9515531Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9515535Z 2025-11-03T16:37:00.9515631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9515826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9515887Z res = mod(**inputs) 2025-11-03T16:37:00.9516154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9516219Z outputs = self.mobilebert( 2025-11-03T16:37:00.9516486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9516555Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9516836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9516912Z layer_outputs = layer_module( 2025-11-03T16:37:00.9517170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9517263Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9517519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9517621Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9517889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9517996Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9517999Z 2025-11-03T16:37:00.9518128Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9518312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9518381Z res = mod(**inputs) 2025-11-03T16:37:00.9518638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9518702Z outputs = self.mobilebert( 2025-11-03T16:37:00.9518966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9519032Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9519296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9519361Z layer_outputs = layer_module( 2025-11-03T16:37:00.9519622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9519718Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9519996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9520124Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9520380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9520467Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9520470Z 2025-11-03T16:37:00.9520563Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9520743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9520813Z res = mod(**inputs) 2025-11-03T16:37:00.9521069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9521142Z outputs = self.mobilebert( 2025-11-03T16:37:00.9521425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9521492Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9521754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9521819Z layer_outputs = layer_module( 2025-11-03T16:37:00.9522082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9522169Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9522433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9522584Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9522842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9522967Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9523221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9523315Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9523318Z 2025-11-03T16:37:00.9523411Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9523601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9523660Z res = mod(**inputs) 2025-11-03T16:37:00.9523930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9524003Z outputs = self.mobilebert( 2025-11-03T16:37:00.9524264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9524340Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9524600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9524664Z layer_outputs = layer_module( 2025-11-03T16:37:00.9524933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9525018Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9525286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9525392Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9525659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9525752Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9525756Z 2025-11-03T16:37:00.9525848Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9526040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9526098Z res = mod(**inputs) 2025-11-03T16:37:00.9526362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9526427Z outputs = self.mobilebert( 2025-11-03T16:37:00.9526685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9526761Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9527020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9527109Z layer_outputs = layer_module( 2025-11-03T16:37:00.9527366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9527459Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9527715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9527820Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9528085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9528191Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9528194Z 2025-11-03T16:37:00.9528309Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9528493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9528555Z res = mod(**inputs) 2025-11-03T16:37:00.9528822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9528887Z outputs = self.mobilebert( 2025-11-03T16:37:00.9529155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9529222Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9529491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9529554Z layer_outputs = layer_module( 2025-11-03T16:37:00.9529829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9529923Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9530185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9530305Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9530561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9530636Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9530647Z 2025-11-03T16:37:00.9530740Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9530922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9530989Z res = mod(**inputs) 2025-11-03T16:37:00.9531249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9531321Z outputs = self.mobilebert( 2025-11-03T16:37:00.9531592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9531659Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9531922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9531986Z layer_outputs = layer_module( 2025-11-03T16:37:00.9532249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9532333Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9532591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9532712Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9532987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9533109Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9533367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9533458Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9533461Z 2025-11-03T16:37:00.9533557Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9533740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9533807Z res = mod(**inputs) 2025-11-03T16:37:00.9534081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9534155Z outputs = self.mobilebert( 2025-11-03T16:37:00.9534419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9534487Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9534757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9534822Z layer_outputs = layer_module( 2025-11-03T16:37:00.9535092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9535178Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9535462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9535567Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9535826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9535913Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9535917Z 2025-11-03T16:37:00.9536011Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9536200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9536258Z res = mod(**inputs) 2025-11-03T16:37:00.9536515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9536588Z outputs = self.mobilebert( 2025-11-03T16:37:00.9536844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9536922Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9537178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9537276Z layer_outputs = layer_module( 2025-11-03T16:37:00.9537536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9537622Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9537897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9538003Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9538270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9538376Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9538381Z 2025-11-03T16:37:00.9538482Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9538679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9538740Z res = mod(**inputs) 2025-11-03T16:37:00.9539004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9539067Z outputs = self.mobilebert( 2025-11-03T16:37:00.9539330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9539395Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9539652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9539723Z layer_outputs = layer_module( 2025-11-03T16:37:00.9539996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9540090Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9540347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9540468Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9540726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9540804Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9540807Z 2025-11-03T16:37:00.9540909Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9541088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9541170Z res = mod(**inputs) 2025-11-03T16:37:00.9541429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9541495Z outputs = self.mobilebert( 2025-11-03T16:37:00.9541758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9541824Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9542086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9542149Z layer_outputs = layer_module( 2025-11-03T16:37:00.9542411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9542496Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9542756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9542879Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9543154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9543275Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9543532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9543618Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9543629Z 2025-11-03T16:37:00.9543722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9543903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9543970Z res = mod(**inputs) 2025-11-03T16:37:00.9544233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9544323Z outputs = self.mobilebert( 2025-11-03T16:37:00.9544587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9544653Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9544925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9544989Z layer_outputs = layer_module( 2025-11-03T16:37:00.9545260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9545374Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9545656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9545743Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9545747Z 2025-11-03T16:37:00.9545843Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9546036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9546094Z res = mod(**inputs) 2025-11-03T16:37:00.9546360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9546424Z outputs = self.mobilebert( 2025-11-03T16:37:00.9546681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9546755Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9547027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9547103Z layer_outputs = layer_module( 2025-11-03T16:37:00.9547363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9547478Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9547744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9547847Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9547851Z 2025-11-03T16:37:00.9547952Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9548135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9548199Z res = mod(**inputs) 2025-11-03T16:37:00.9548456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9548522Z outputs = self.mobilebert( 2025-11-03T16:37:00.9548791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9548904Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9549171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9549236Z layer_outputs = layer_module( 2025-11-03T16:37:00.9549495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9549655Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9549916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9550017Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9550020Z 2025-11-03T16:37:00.9550117Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9550324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9550383Z res = mod(**inputs) 2025-11-03T16:37:00.9550641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9550714Z outputs = self.mobilebert( 2025-11-03T16:37:00.9550975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9551052Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9551310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9551379Z layer_outputs = layer_module( 2025-11-03T16:37:00.9551663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9551814Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9552087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9552201Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9552463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9552548Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9552551Z 2025-11-03T16:37:00.9552645Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9552849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9552912Z res = mod(**inputs) 2025-11-03T16:37:00.9553176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9553244Z outputs = self.mobilebert( 2025-11-03T16:37:00.9553508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9553576Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9553836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9553912Z layer_outputs = layer_module( 2025-11-03T16:37:00.9554241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9554396Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9554675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9554821Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9555105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9555189Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9555193Z 2025-11-03T16:37:00.9555301Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9555496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9555557Z res = mod(**inputs) 2025-11-03T16:37:00.9555850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9555917Z outputs = self.mobilebert( 2025-11-03T16:37:00.9556194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9556278Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9556550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9556626Z layer_outputs = layer_module( 2025-11-03T16:37:00.9556937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9557094Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9557379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9557506Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9557798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9557922Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9558200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9558290Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9558293Z 2025-11-03T16:37:00.9558399Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9558591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9558652Z res = mod(**inputs) 2025-11-03T16:37:00.9558931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9559027Z outputs = self.mobilebert( 2025-11-03T16:37:00.9559312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9559384Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9559665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9559733Z layer_outputs = layer_module( 2025-11-03T16:37:00.9560006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9560175Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9560449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9560566Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9560842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9560940Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9560951Z 2025-11-03T16:37:00.9561051Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9561243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9561312Z res = mod(**inputs) 2025-11-03T16:37:00.9561584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9561656Z outputs = self.mobilebert( 2025-11-03T16:37:00.9561929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9561998Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9562283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9562375Z layer_outputs = layer_module( 2025-11-03T16:37:00.9562655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9562738Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9563009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9563084Z self_outputs = self.self( 2025-11-03T16:37:00.9563356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9563431Z self.value(value_tensor) 2025-11-03T16:37:00.9563434Z 2025-11-03T16:37:00.9563533Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9563749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9563812Z res = mod(**inputs) 2025-11-03T16:37:00.9564093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9564168Z outputs = self.mobilebert( 2025-11-03T16:37:00.9564446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9564522Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9564800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9564864Z layer_outputs = layer_module( 2025-11-03T16:37:00.9565130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9565294Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9565560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9565662Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9565923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9565999Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9566003Z 2025-11-03T16:37:00.9566095Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9566281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9566339Z res = mod(**inputs) 2025-11-03T16:37:00.9566604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9566669Z outputs = self.mobilebert( 2025-11-03T16:37:00.9566925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9567016Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9567272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9567343Z layer_outputs = layer_module( 2025-11-03T16:37:00.9567600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9567755Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9568015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9568118Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9568383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9568481Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9568742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9568825Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9568829Z 2025-11-03T16:37:00.9568931Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9569112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9569170Z res = mod(**inputs) 2025-11-03T16:37:00.9569433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9569514Z outputs = self.mobilebert( 2025-11-03T16:37:00.9569775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9569844Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9570126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9570199Z layer_outputs = layer_module( 2025-11-03T16:37:00.9570463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9570551Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9570816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9570881Z self_outputs = self.self( 2025-11-03T16:37:00.9571171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9571240Z self.query(query_tensor) 2025-11-03T16:37:00.9571244Z 2025-11-03T16:37:00.9571350Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9571535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9571601Z res = mod(**inputs) 2025-11-03T16:37:00.9571866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9571930Z outputs = self.mobilebert( 2025-11-03T16:37:00.9572201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9572268Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9572541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9572607Z layer_outputs = layer_module( 2025-11-03T16:37:00.9572874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9572977Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9573241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9573314Z self_outputs = self.self( 2025-11-03T16:37:00.9573581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9573652Z self.key(key_tensor) 2025-11-03T16:37:00.9573656Z 2025-11-03T16:37:00.9573735Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9573809Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9573916Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9574103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9574188Z res = mod(**inputs) 2025-11-03T16:37:00.9574454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9574518Z outputs = self.mobilebert( 2025-11-03T16:37:00.9574795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9574864Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9575135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9575201Z layer_outputs = layer_module( 2025-11-03T16:37:00.9575481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9575568Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9575833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9575961Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9576226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9576313Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9576317Z 2025-11-03T16:37:00.9576413Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9576600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9576667Z res = mod(**inputs) 2025-11-03T16:37:00.9576944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9577019Z outputs = self.mobilebert( 2025-11-03T16:37:00.9577282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9577350Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9577616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9577681Z layer_outputs = layer_module( 2025-11-03T16:37:00.9577950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9578027Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9578300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9578415Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9578676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9579146Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9579411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9579503Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9579507Z 2025-11-03T16:37:00.9579604Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9579798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9579857Z res = mod(**inputs) 2025-11-03T16:37:00.9580124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9580200Z outputs = self.mobilebert( 2025-11-03T16:37:00.9580467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9580563Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9580827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9580893Z layer_outputs = layer_module( 2025-11-03T16:37:00.9581166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9581256Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9581528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9581637Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9581937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9582022Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9582026Z 2025-11-03T16:37:00.9582120Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9582317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9582375Z res = mod(**inputs) 2025-11-03T16:37:00.9582655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9582723Z outputs = self.mobilebert( 2025-11-03T16:37:00.9582994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9583072Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9583358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9583438Z layer_outputs = layer_module( 2025-11-03T16:37:00.9583717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9583816Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9584080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9584185Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9584465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9584571Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9584575Z 2025-11-03T16:37:00.9584683Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9584870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9584950Z res = mod(**inputs) 2025-11-03T16:37:00.9585231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9585297Z outputs = self.mobilebert( 2025-11-03T16:37:00.9585578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9585649Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9585930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9585997Z layer_outputs = layer_module( 2025-11-03T16:37:00.9586274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9586370Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9586657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9586783Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9587048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9587126Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9587137Z 2025-11-03T16:37:00.9587231Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9587417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9587482Z res = mod(**inputs) 2025-11-03T16:37:00.9587763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9587839Z outputs = self.mobilebert( 2025-11-03T16:37:00.9588108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9588176Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9588448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9588514Z layer_outputs = layer_module( 2025-11-03T16:37:00.9588786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9588873Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9589154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9589284Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9589548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9589673Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9589937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9590030Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9590034Z 2025-11-03T16:37:00.9590129Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9590319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9590386Z res = mod(**inputs) 2025-11-03T16:37:00.9590655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9590728Z outputs = self.mobilebert( 2025-11-03T16:37:00.9590996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9591084Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9591356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9591421Z layer_outputs = layer_module( 2025-11-03T16:37:00.9591692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9591779Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9592049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9592155Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9592421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9592524Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9592528Z 2025-11-03T16:37:00.9592623Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9592817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9592878Z res = mod(**inputs) 2025-11-03T16:37:00.9593143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9593218Z outputs = self.mobilebert( 2025-11-03T16:37:00.9593483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9593561Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9593847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9593927Z layer_outputs = layer_module( 2025-11-03T16:37:00.9594278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9594378Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9594668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9594778Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9595122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9595243Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9595265Z 2025-11-03T16:37:00.9595375Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9595566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9595629Z res = mod(**inputs) 2025-11-03T16:37:00.9595906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9595973Z outputs = self.mobilebert( 2025-11-03T16:37:00.9596249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9596317Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9596585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9596660Z layer_outputs = layer_module( 2025-11-03T16:37:00.9596939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9597036Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9597327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9597450Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9597707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9597785Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9597789Z 2025-11-03T16:37:00.9597891Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9598073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9598141Z res = mod(**inputs) 2025-11-03T16:37:00.9598400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9598481Z outputs = self.mobilebert( 2025-11-03T16:37:00.9598757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9598824Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9599098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9599163Z layer_outputs = layer_module( 2025-11-03T16:37:00.9599440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9599526Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9599810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9599938Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9600197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9600319Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9600581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9600665Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9600677Z 2025-11-03T16:37:00.9600771Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9600953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9601021Z res = mod(**inputs) 2025-11-03T16:37:00.9601299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9601373Z outputs = self.mobilebert( 2025-11-03T16:37:00.9601631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9601700Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9601964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9602027Z layer_outputs = layer_module( 2025-11-03T16:37:00.9602291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9602376Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9602632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9602743Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9602998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9603097Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9603101Z 2025-11-03T16:37:00.9603194Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9603382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9603440Z res = mod(**inputs) 2025-11-03T16:37:00.9603697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9603771Z outputs = self.mobilebert( 2025-11-03T16:37:00.9604028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9604103Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9604358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9604451Z layer_outputs = layer_module( 2025-11-03T16:37:00.9604715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9604800Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9605064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9605166Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9605430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9605533Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9605554Z 2025-11-03T16:37:00.9605650Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9605843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9605905Z res = mod(**inputs) 2025-11-03T16:37:00.9606171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9606240Z outputs = self.mobilebert( 2025-11-03T16:37:00.9606502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9606578Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9606837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9606911Z layer_outputs = layer_module( 2025-11-03T16:37:00.9607186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9607280Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9607536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9607649Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9607913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9607989Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9607992Z 2025-11-03T16:37:00.9608091Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9608274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9608334Z res = mod(**inputs) 2025-11-03T16:37:00.9608600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9608683Z outputs = self.mobilebert( 2025-11-03T16:37:00.9608949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9609015Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9609281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9609345Z layer_outputs = layer_module( 2025-11-03T16:37:00.9609603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9609695Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9609958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9610080Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9610361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9610479Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9610743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9610828Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9610831Z 2025-11-03T16:37:00.9610933Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9611120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9611185Z res = mod(**inputs) 2025-11-03T16:37:00.9611468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9611538Z outputs = self.mobilebert( 2025-11-03T16:37:00.9611822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9611889Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9612159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9612225Z layer_outputs = layer_module( 2025-11-03T16:37:00.9612498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9612613Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9612899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9612987Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9612992Z 2025-11-03T16:37:00.9613092Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9613452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9613519Z res = mod(**inputs) 2025-11-03T16:37:00.9613798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9613874Z outputs = self.mobilebert( 2025-11-03T16:37:00.9614148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9614229Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9614506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9614593Z layer_outputs = layer_module( 2025-11-03T16:37:00.9614866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9615027Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9615289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9615403Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9615407Z 2025-11-03T16:37:00.9615502Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9615691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9615757Z res = mod(**inputs) 2025-11-03T16:37:00.9616023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9616101Z outputs = self.mobilebert( 2025-11-03T16:37:00.9616365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9616469Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9616736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9616802Z layer_outputs = layer_module( 2025-11-03T16:37:00.9617080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9617231Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9617503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9617622Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9617626Z 2025-11-03T16:37:00.9617722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9617919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9617979Z res = mod(**inputs) 2025-11-03T16:37:00.9618252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9618316Z outputs = self.mobilebert( 2025-11-03T16:37:00.9618589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9618658Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9618926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9619025Z layer_outputs = layer_module( 2025-11-03T16:37:00.9619290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9619446Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9619710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9619827Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9620099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9620185Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9620189Z 2025-11-03T16:37:00.9620294Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9620480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9620548Z res = mod(**inputs) 2025-11-03T16:37:00.9620812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9620896Z outputs = self.mobilebert( 2025-11-03T16:37:00.9621165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9621231Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9621501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9621568Z layer_outputs = layer_module( 2025-11-03T16:37:00.9621831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9621985Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9622251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9622394Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9622657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9622751Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9622755Z 2025-11-03T16:37:00.9622853Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9623045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9623104Z res = mod(**inputs) 2025-11-03T16:37:00.9623378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9623445Z outputs = self.mobilebert( 2025-11-03T16:37:00.9623727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9623805Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9624071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9624144Z layer_outputs = layer_module( 2025-11-03T16:37:00.9624408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9624555Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9624827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9624959Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9625234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9625353Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9625626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9625711Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9625714Z 2025-11-03T16:37:00.9625811Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9626007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9626065Z res = mod(**inputs) 2025-11-03T16:37:00.9626337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9626403Z outputs = self.mobilebert( 2025-11-03T16:37:00.9626669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9626770Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9627034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9627109Z layer_outputs = layer_module( 2025-11-03T16:37:00.9627373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9627531Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9627794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9627897Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9628172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9628267Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9628271Z 2025-11-03T16:37:00.9628373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9628557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9628616Z res = mod(**inputs) 2025-11-03T16:37:00.9628883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9628947Z outputs = self.mobilebert( 2025-11-03T16:37:00.9629224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9629291Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9629577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9629644Z layer_outputs = layer_module( 2025-11-03T16:37:00.9629909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9629997Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9630295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9630370Z self_outputs = self.self( 2025-11-03T16:37:00.9630636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9630703Z self.value(value_tensor) 2025-11-03T16:37:00.9630713Z 2025-11-03T16:37:00.9630809Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9631013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9631081Z res = mod(**inputs) 2025-11-03T16:37:00.9631347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9631423Z outputs = self.mobilebert( 2025-11-03T16:37:00.9631693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9631761Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9632033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9632100Z layer_outputs = layer_module( 2025-11-03T16:37:00.9632372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9632527Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9632792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9632923Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9633189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9633274Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9633278Z 2025-11-03T16:37:00.9633373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9633568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9633628Z res = mod(**inputs) 2025-11-03T16:37:00.9633894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9633970Z outputs = self.mobilebert( 2025-11-03T16:37:00.9634303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9634409Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9634675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9634743Z layer_outputs = layer_module( 2025-11-03T16:37:00.9635023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9635180Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9635480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9635599Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9635876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9635961Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9636231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9636328Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9636331Z 2025-11-03T16:37:00.9636429Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9636628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9636689Z res = mod(**inputs) 2025-11-03T16:37:00.9636970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9637050Z outputs = self.mobilebert( 2025-11-03T16:37:00.9637310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9637390Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9637649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9637722Z layer_outputs = layer_module( 2025-11-03T16:37:00.9637983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9638066Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9638334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9638399Z self_outputs = self.self( 2025-11-03T16:37:00.9638667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9638732Z self.query(query_tensor) 2025-11-03T16:37:00.9638750Z 2025-11-03T16:37:00.9638851Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9639031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9639089Z res = mod(**inputs) 2025-11-03T16:37:00.9639355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9639418Z outputs = self.mobilebert( 2025-11-03T16:37:00.9639682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9639747Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9640006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9640077Z layer_outputs = layer_module( 2025-11-03T16:37:00.9640353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9640440Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9640697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9640760Z self_outputs = self.self( 2025-11-03T16:37:00.9641023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9641086Z self.key(key_tensor) 2025-11-03T16:37:00.9641089Z 2025-11-03T16:37:00.9641173Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9641246Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9641348Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9641544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9641607Z res = mod(**inputs) 2025-11-03T16:37:00.9641878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9641941Z outputs = self.mobilebert( 2025-11-03T16:37:00.9642204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9642270Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9642525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9642598Z layer_outputs = layer_module( 2025-11-03T16:37:00.9642894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9642978Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9643238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9643361Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9643618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9643695Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9643699Z 2025-11-03T16:37:00.9643969Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9644151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9644216Z res = mod(**inputs) 2025-11-03T16:37:00.9644477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9644543Z outputs = self.mobilebert( 2025-11-03T16:37:00.9644808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9644889Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9645152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9645215Z layer_outputs = layer_module( 2025-11-03T16:37:00.9645478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9645553Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9645810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9645934Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9646189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9646327Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9646582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9646671Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9646675Z 2025-11-03T16:37:00.9646766Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9646944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9647008Z res = mod(**inputs) 2025-11-03T16:37:00.9647263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9647352Z outputs = self.mobilebert( 2025-11-03T16:37:00.9647614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9647682Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9647947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9648012Z layer_outputs = layer_module( 2025-11-03T16:37:00.9648280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9648366Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9648632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9648759Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9649020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9649107Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9649110Z 2025-11-03T16:37:00.9649204Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9649391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9649450Z res = mod(**inputs) 2025-11-03T16:37:00.9649706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9649779Z outputs = self.mobilebert( 2025-11-03T16:37:00.9650034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9650108Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9650367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9650441Z layer_outputs = layer_module( 2025-11-03T16:37:00.9650711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9650798Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9651062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9651165Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9651427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9651530Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9651533Z 2025-11-03T16:37:00.9651630Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9651817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9651891Z res = mod(**inputs) 2025-11-03T16:37:00.9652157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9652221Z outputs = self.mobilebert( 2025-11-03T16:37:00.9652482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9652548Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9652804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9652879Z layer_outputs = layer_module( 2025-11-03T16:37:00.9653152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9653246Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9653509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9653624Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9653892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9653968Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9653971Z 2025-11-03T16:37:00.9654071Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9654252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9654316Z res = mod(**inputs) 2025-11-03T16:37:00.9654590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9654656Z outputs = self.mobilebert( 2025-11-03T16:37:00.9654925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9654991Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9655255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9655318Z layer_outputs = layer_module( 2025-11-03T16:37:00.9655575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9655666Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9655922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9656045Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9656301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9656436Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9656694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9656778Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9656781Z 2025-11-03T16:37:00.9656882Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9657064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9657129Z res = mod(**inputs) 2025-11-03T16:37:00.9657390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9657453Z outputs = self.mobilebert( 2025-11-03T16:37:00.9657724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9657811Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9658073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9658136Z layer_outputs = layer_module( 2025-11-03T16:37:00.9658399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9658484Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9658742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9658866Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9659121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9659212Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9659215Z 2025-11-03T16:37:00.9659311Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9659502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9659564Z res = mod(**inputs) 2025-11-03T16:37:00.9659823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9659897Z outputs = self.mobilebert( 2025-11-03T16:37:00.9660154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9660243Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9660500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9660566Z layer_outputs = layer_module( 2025-11-03T16:37:00.9660830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9660915Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9661186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9661289Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9661552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9661665Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9661670Z 2025-11-03T16:37:00.9661768Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9661964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9662040Z res = mod(**inputs) 2025-11-03T16:37:00.9662317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9662383Z outputs = self.mobilebert( 2025-11-03T16:37:00.9662652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9662726Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9662996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9663070Z layer_outputs = layer_module( 2025-11-03T16:37:00.9663341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9663428Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9663719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9663836Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9664109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9664188Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9664191Z 2025-11-03T16:37:00.9664293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9664481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9664539Z res = mod(**inputs) 2025-11-03T16:37:00.9664834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9664903Z outputs = self.mobilebert( 2025-11-03T16:37:00.9665179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9665246Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9665516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9665579Z layer_outputs = layer_module( 2025-11-03T16:37:00.9665842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9665936Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9666220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9666348Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9666616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9666732Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9667007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9667091Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9667095Z 2025-11-03T16:37:00.9667196Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9667386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9667450Z res = mod(**inputs) 2025-11-03T16:37:00.9667720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9667786Z outputs = self.mobilebert( 2025-11-03T16:37:00.9668059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9668142Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9668416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9668481Z layer_outputs = layer_module( 2025-11-03T16:37:00.9668747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9668841Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9669110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9669222Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9669488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9669591Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9669595Z 2025-11-03T16:37:00.9669689Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9669876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9669943Z res = mod(**inputs) 2025-11-03T16:37:00.9670211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9670283Z outputs = self.mobilebert( 2025-11-03T16:37:00.9670550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9670642Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9670918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9670989Z layer_outputs = layer_module( 2025-11-03T16:37:00.9671269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9671360Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9671639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9671746Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9672021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9672151Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9672157Z 2025-11-03T16:37:00.9672255Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9672454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9672516Z res = mod(**inputs) 2025-11-03T16:37:00.9672793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9672867Z outputs = self.mobilebert( 2025-11-03T16:37:00.9673138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9673211Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9673484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9673555Z layer_outputs = layer_module( 2025-11-03T16:37:00.9673894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9673990Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9674386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9674516Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9674833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9674922Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9674926Z 2025-11-03T16:37:00.9675041Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9675253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9675322Z res = mod(**inputs) 2025-11-03T16:37:00.9675637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9675724Z outputs = self.mobilebert( 2025-11-03T16:37:00.9675999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9676068Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9676346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9676420Z layer_outputs = layer_module( 2025-11-03T16:37:00.9676684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9676780Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9677060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9677179Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9677453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9677569Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9677840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9677925Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9677929Z 2025-11-03T16:37:00.9678031Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9678217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9678276Z res = mod(**inputs) 2025-11-03T16:37:00.9678567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9678635Z outputs = self.mobilebert( 2025-11-03T16:37:00.9678913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9678982Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9679256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9679332Z layer_outputs = layer_module( 2025-11-03T16:37:00.9679639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9679774Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9680083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9680179Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9680183Z 2025-11-03T16:37:00.9680293Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9680516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9680589Z res = mod(**inputs) 2025-11-03T16:37:00.9680892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9680973Z outputs = self.mobilebert( 2025-11-03T16:37:00.9681275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9681358Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9681662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9681736Z layer_outputs = layer_module( 2025-11-03T16:37:00.9682038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9682184Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9682489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9682606Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9682610Z 2025-11-03T16:37:00.9682719Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9682934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9682999Z res = mod(**inputs) 2025-11-03T16:37:00.9683314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9683405Z outputs = self.mobilebert( 2025-11-03T16:37:00.9683711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9683790Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9684090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9684172Z layer_outputs = layer_module( 2025-11-03T16:37:00.9684475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9684650Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9684947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9685065Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9685075Z 2025-11-03T16:37:00.9685184Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9685393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9685464Z res = mod(**inputs) 2025-11-03T16:37:00.9685768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9685846Z outputs = self.mobilebert( 2025-11-03T16:37:00.9686140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9686212Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9686517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9686591Z layer_outputs = layer_module( 2025-11-03T16:37:00.9686896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9687082Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9687387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9687524Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9687830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9687930Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9687934Z 2025-11-03T16:37:00.9688039Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9688259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9688328Z res = mod(**inputs) 2025-11-03T16:37:00.9688586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9688674Z outputs = self.mobilebert( 2025-11-03T16:37:00.9688933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9689004Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9689261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9689324Z layer_outputs = layer_module( 2025-11-03T16:37:00.9689601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9689748Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9690032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9690148Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9690408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9690484Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9690488Z 2025-11-03T16:37:00.9690580Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9690766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9690823Z res = mod(**inputs) 2025-11-03T16:37:00.9691080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9691156Z outputs = self.mobilebert( 2025-11-03T16:37:00.9691426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9691494Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9691751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9691821Z layer_outputs = layer_module( 2025-11-03T16:37:00.9692081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9692229Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9692489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9692605Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9692872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9693002Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9693265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9693349Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9693352Z 2025-11-03T16:37:00.9693452Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9693636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9693694Z res = mod(**inputs) 2025-11-03T16:37:00.9693966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9694032Z outputs = self.mobilebert( 2025-11-03T16:37:00.9694303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9694399Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9694668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9694742Z layer_outputs = layer_module( 2025-11-03T16:37:00.9695007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9695168Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9695439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9695551Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9695837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9695917Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9695922Z 2025-11-03T16:37:00.9696028Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9696215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9696282Z res = mod(**inputs) 2025-11-03T16:37:00.9696559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9696624Z outputs = self.mobilebert( 2025-11-03T16:37:00.9696895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9696964Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9697255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9697324Z layer_outputs = layer_module( 2025-11-03T16:37:00.9697599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9697682Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9697950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9698025Z self_outputs = self.self( 2025-11-03T16:37:00.9698301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-11-03T16:37:00.9698379Z self.value(value_tensor) 2025-11-03T16:37:00.9698382Z 2025-11-03T16:37:00.9698484Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9698684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9698756Z res = mod(**inputs) 2025-11-03T16:37:00.9699041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9699134Z outputs = self.mobilebert( 2025-11-03T16:37:00.9699413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9699489Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9699768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9699837Z layer_outputs = layer_module( 2025-11-03T16:37:00.9700129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9700285Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9700567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-11-03T16:37:00.9700692Z bottlenecked_hidden_states = self.input(hidden_states) 2025-11-03T16:37:00.9700973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-11-03T16:37:00.9701062Z layer_input = self.dense(hidden_states) 2025-11-03T16:37:00.9701065Z 2025-11-03T16:37:00.9701165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9701367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9701429Z res = mod(**inputs) 2025-11-03T16:37:00.9701713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9701799Z outputs = self.mobilebert( 2025-11-03T16:37:00.9702078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9702159Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9702435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9702512Z layer_outputs = layer_module( 2025-11-03T16:37:00.9702788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-11-03T16:37:00.9702946Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-11-03T16:37:00.9703237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-11-03T16:37:00.9703361Z shared_attention_input = self.attention(hidden_states) 2025-11-03T16:37:00.9703647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-11-03T16:37:00.9703746Z layer_input = self.LayerNorm(layer_input) 2025-11-03T16:37:00.9704021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9704108Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9704112Z 2025-11-03T16:37:00.9704210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9704405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9704466Z res = mod(**inputs) 2025-11-03T16:37:00.9704744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9704813Z outputs = self.mobilebert( 2025-11-03T16:37:00.9705094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9705183Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9705464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9705541Z layer_outputs = layer_module( 2025-11-03T16:37:00.9705824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9705916Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9706201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9706271Z self_outputs = self.self( 2025-11-03T16:37:00.9706565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-11-03T16:37:00.9706635Z self.query(query_tensor) 2025-11-03T16:37:00.9706655Z 2025-11-03T16:37:00.9706762Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9706965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9707034Z res = mod(**inputs) 2025-11-03T16:37:00.9707314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9707382Z outputs = self.mobilebert( 2025-11-03T16:37:00.9707669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9707739Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9708047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9708118Z layer_outputs = layer_module( 2025-11-03T16:37:00.9708396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9708488Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9708766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-11-03T16:37:00.9708843Z self_outputs = self.self( 2025-11-03T16:37:00.9709124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-11-03T16:37:00.9709190Z self.key(key_tensor) 2025-11-03T16:37:00.9709200Z 2025-11-03T16:37:00.9709283Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9709362Z cudagraph partition due to non gpu ops 2025-11-03T16:37:00.9709485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9709684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9709758Z res = mod(**inputs) 2025-11-03T16:37:00.9710037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9710106Z outputs = self.mobilebert( 2025-11-03T16:37:00.9710394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9710471Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9710787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9710858Z layer_outputs = layer_module( 2025-11-03T16:37:00.9711167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9711265Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9711558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9711712Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9712012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-11-03T16:37:00.9712110Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9712114Z 2025-11-03T16:37:00.9712221Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9712428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9712503Z res = mod(**inputs) 2025-11-03T16:37:00.9712810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9712887Z outputs = self.mobilebert( 2025-11-03T16:37:00.9713300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9713450Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9713766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9713843Z layer_outputs = layer_module( 2025-11-03T16:37:00.9714241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-11-03T16:37:00.9714340Z self_attention_outputs = self.attention( 2025-11-03T16:37:00.9714654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-11-03T16:37:00.9714822Z attention_output = self.output(self_outputs[0], layer_input) 2025-11-03T16:37:00.9715130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-11-03T16:37:00.9715263Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9715535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9715630Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9715634Z 2025-11-03T16:37:00.9715734Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9715930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9716002Z res = mod(**inputs) 2025-11-03T16:37:00.9716294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9716371Z outputs = self.mobilebert( 2025-11-03T16:37:00.9716638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9716718Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9716993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9717059Z layer_outputs = layer_module( 2025-11-03T16:37:00.9717338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9717430Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9717705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9717813Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9718086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9718205Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9718208Z 2025-11-03T16:37:00.9718306Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9718508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9718569Z res = mod(**inputs) 2025-11-03T16:37:00.9718846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9718913Z outputs = self.mobilebert( 2025-11-03T16:37:00.9719184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9719261Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9719535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9719625Z layer_outputs = layer_module( 2025-11-03T16:37:00.9719895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9719986Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9720266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9720371Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9720648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9720756Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9720759Z 2025-11-03T16:37:00.9720882Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9721084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9721145Z res = mod(**inputs) 2025-11-03T16:37:00.9721414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9721479Z outputs = self.mobilebert( 2025-11-03T16:37:00.9721750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9721818Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9722080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9722153Z layer_outputs = layer_module( 2025-11-03T16:37:00.9722442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9722536Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9722802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9722925Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9723178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9723255Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9723258Z 2025-11-03T16:37:00.9723357Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9723538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9723601Z res = mod(**inputs) 2025-11-03T16:37:00.9723856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9723920Z outputs = self.mobilebert( 2025-11-03T16:37:00.9724186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9724267Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9724532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9724597Z layer_outputs = layer_module( 2025-11-03T16:37:00.9724861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9724948Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9725207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9725331Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9725587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9725724Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9725988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9726078Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9726081Z 2025-11-03T16:37:00.9726175Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9726359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9726425Z res = mod(**inputs) 2025-11-03T16:37:00.9726685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9726770Z outputs = self.mobilebert( 2025-11-03T16:37:00.9727025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9727093Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9727357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9727422Z layer_outputs = layer_module( 2025-11-03T16:37:00.9727684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9727769Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9728032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9728149Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9728410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9728496Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9728499Z 2025-11-03T16:37:00.9728592Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9728782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9728841Z res = mod(**inputs) 2025-11-03T16:37:00.9729097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9729167Z outputs = self.mobilebert( 2025-11-03T16:37:00.9729428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9729502Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9729759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9729849Z layer_outputs = layer_module( 2025-11-03T16:37:00.9730107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9730192Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9730456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9730557Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9730821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9730922Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9730927Z 2025-11-03T16:37:00.9731021Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9731207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9731285Z res = mod(**inputs) 2025-11-03T16:37:00.9731553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9731616Z outputs = self.mobilebert( 2025-11-03T16:37:00.9731883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9731948Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9732206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9732279Z layer_outputs = layer_module( 2025-11-03T16:37:00.9732552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9732646Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9732907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9733020Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9733283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9733360Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9733363Z 2025-11-03T16:37:00.9733464Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9733645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9733711Z res = mod(**inputs) 2025-11-03T16:37:00.9733983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9734052Z outputs = self.mobilebert( 2025-11-03T16:37:00.9734318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9734383Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9734651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9734716Z layer_outputs = layer_module( 2025-11-03T16:37:00.9734972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9735064Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9735329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9735447Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9735705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9735836Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9736096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9736180Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9736183Z 2025-11-03T16:37:00.9736284Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9736463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9736527Z res = mod(**inputs) 2025-11-03T16:37:00.9736784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9736848Z outputs = self.mobilebert( 2025-11-03T16:37:00.9737127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9737194Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9737455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9737520Z layer_outputs = layer_module( 2025-11-03T16:37:00.9737782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9737867Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9738121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9738269Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9738528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9738614Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9738617Z 2025-11-03T16:37:00.9738709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9738896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9738954Z res = mod(**inputs) 2025-11-03T16:37:00.9739210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9739281Z outputs = self.mobilebert( 2025-11-03T16:37:00.9739541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9739636Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9739896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9739963Z layer_outputs = layer_module( 2025-11-03T16:37:00.9740228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9740313Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9740581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-11-03T16:37:00.9740681Z intermediate_output = self.intermediate(hidden_states) 2025-11-03T16:37:00.9740939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9741051Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9741054Z 2025-11-03T16:37:00.9741150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9741342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9741414Z res = mod(**inputs) 2025-11-03T16:37:00.9741682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9741746Z outputs = self.mobilebert( 2025-11-03T16:37:00.9742011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9742088Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9742354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9742428Z layer_outputs = layer_module( 2025-11-03T16:37:00.9742702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9742802Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9743072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9743186Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9743454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-11-03T16:37:00.9743529Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9743532Z 2025-11-03T16:37:00.9743631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9743814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9743871Z res = mod(**inputs) 2025-11-03T16:37:00.9744151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9744218Z outputs = self.mobilebert( 2025-11-03T16:37:00.9744487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9744553Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9744823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9744888Z layer_outputs = layer_module( 2025-11-03T16:37:00.9745149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-11-03T16:37:00.9745243Z attention_output = ffn_module(attention_output) 2025-11-03T16:37:00.9745516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-11-03T16:37:00.9745639Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-11-03T16:37:00.9745900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-11-03T16:37:00.9746012Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9746277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9746359Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9746363Z 2025-11-03T16:37:00.9746463Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9746646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9746710Z res = mod(**inputs) 2025-11-03T16:37:00.9746968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9747031Z outputs = self.mobilebert( 2025-11-03T16:37:00.9747313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9747380Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9747643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9747708Z layer_outputs = layer_module( 2025-11-03T16:37:00.9747964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9748083Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9748341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-11-03T16:37:00.9748427Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9748430Z 2025-11-03T16:37:00.9748540Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9748732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9748790Z res = mod(**inputs) 2025-11-03T16:37:00.9749047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9749119Z outputs = self.mobilebert( 2025-11-03T16:37:00.9749377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9749450Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9749710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9749788Z layer_outputs = layer_module( 2025-11-03T16:37:00.9750055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-11-03T16:37:00.9750167Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:37:00.9750433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-11-03T16:37:00.9750532Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:37:00.9750535Z 2025-11-03T16:37:00.9750633Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9750816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9750873Z res = mod(**inputs) 2025-11-03T16:37:00.9751153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9751219Z outputs = self.mobilebert( 2025-11-03T16:37:00.9751481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9751549Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9751804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9751875Z layer_outputs = layer_module( 2025-11-03T16:37:00.9752132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9752283Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9752540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-11-03T16:37:00.9752632Z layer_output = self.dense(intermediate_states) 2025-11-03T16:37:00.9752637Z 2025-11-03T16:37:00.9752729Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9752926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9752993Z res = mod(**inputs) 2025-11-03T16:37:00.9753251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9753320Z outputs = self.mobilebert( 2025-11-03T16:37:00.9753579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9753652Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9753916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9753984Z layer_outputs = layer_module( 2025-11-03T16:37:00.9754322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9754505Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9754811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-11-03T16:37:00.9754946Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-11-03T16:37:00.9755263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9755371Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9755375Z 2025-11-03T16:37:00.9755486Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9755717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9755804Z res = mod(**inputs) 2025-11-03T16:37:00.9756130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9756214Z outputs = self.mobilebert( 2025-11-03T16:37:00.9756533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9756623Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9756926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9757003Z layer_outputs = layer_module( 2025-11-03T16:37:00.9757270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9757439Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9757714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9757833Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9758104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-11-03T16:37:00.9758184Z layer_outputs = self.dense(hidden_states) 2025-11-03T16:37:00.9758187Z 2025-11-03T16:37:00.9758292Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9758480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9758539Z res = mod(**inputs) 2025-11-03T16:37:00.9758810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-11-03T16:37:00.9758878Z outputs = self.mobilebert( 2025-11-03T16:37:00.9759154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-11-03T16:37:00.9759237Z encoder_outputs = self.encoder( 2025-11-03T16:37:00.9759498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-11-03T16:37:00.9759571Z layer_outputs = layer_module( 2025-11-03T16:37:00.9759831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-11-03T16:37:00.9759983Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-11-03T16:37:00.9760248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-11-03T16:37:00.9760369Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-11-03T16:37:00.9760636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-11-03T16:37:00.9760767Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-11-03T16:37:00.9761041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-11-03T16:37:00.9761126Z return input_tensor * self.weight + self.bias 2025-11-03T16:37:00.9761129Z 2025-11-03T16:37:00.9761233Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9761418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9761484Z res = mod(**inputs) 2025-11-03T16:37:00.9761753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-11-03T16:37:00.9761844Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:37:00.9762142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-11-03T16:37:00.9762254Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:37:00.9762527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-11-03T16:37:00.9762615Z hidden_states = self.transform(hidden_states) 2025-11-03T16:37:00.9762880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-11-03T16:37:00.9762964Z hidden_states = self.dense(hidden_states) 2025-11-03T16:37:00.9762967Z 2025-11-03T16:37:00.9763062Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9763257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9763332Z res = mod(**inputs) 2025-11-03T16:37:00.9763607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-11-03T16:37:00.9763694Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:37:00.9763971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-11-03T16:37:00.9764087Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:37:00.9764353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-11-03T16:37:00.9764563Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-11-03T16:37:00.9764566Z 2025-11-03T16:37:00.9764664Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9764859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9764920Z res = mod(**inputs) 2025-11-03T16:37:00.9765186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-11-03T16:37:00.9765297Z prediction_scores = self.cls(sequence_output) 2025-11-03T16:37:00.9765563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-11-03T16:37:00.9765671Z prediction_scores = self.predictions(sequence_output) 2025-11-03T16:37:00.9765937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-11-03T16:37:00.9766013Z hidden_states += self.decoder.bias 2025-11-03T16:37:00.9766016Z 2025-11-03T16:37:00.9766118Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:00.9766305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:00.9766373Z res = mod(**inputs) 2025-11-03T16:37:00.9766638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-11-03T16:37:00.9766840Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:37:00.9766843Z 2025-11-03T16:37:13.7993890Z Compilation time (from dynamo_timed): 37.822801585 2025-11-03T16:37:13.7995490Z pass 2025-11-03T16:37:13.7995928Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:13.8002312Z TIMING: _recursive_pre_grad_passes:0.02078 _recursive_joint_graph_passes:1.79161 _recursive_post_grad_passes:0.16868 async_compile.wait:0.66401 code_gen:10.60743 inductor_compile:14.95056 backend_compile:26.29142 gc:0.00021 entire_frame_compile:37.8228 total_wall_time:37.8228 2025-11-03T16:37:13.8004149Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:30048 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:8820 2025-11-03T16:37:13.8004723Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-11-03T16:37:17.0807990Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:37:17.0808851Z import pynvml # type: ignore[import] 2025-11-03T16:37:20.2203982Z 2025-11-03T16:37:21.7385215Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:37:21.7387940Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:37:21.7394525Z cpu eval OPTForCausalLM 2025-11-03T16:37:23.6989802Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:24.6244018Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:25.5679957Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:32.2383432Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2387835Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2389599Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2389823Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2390029Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2390231Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2395227Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2395556Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2400520Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2405037Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2409147Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2414507Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2417700Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2418209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2420677Z res = mod(**inputs) 2025-11-03T16:37:32.2421137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2421585Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2426600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2427178Z outputs = self.model.decoder( 2025-11-03T16:37:32.2427636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2428422Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2428993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2429413Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2429790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2430372Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2430773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2431174Z return func(*args, **kwargs) 2025-11-03T16:37:32.2431579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2432003Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2432425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2432814Z return func(*args, **kwargs) 2025-11-03T16:37:32.2433255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2433686Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2433864Z 2025-11-03T16:37:32.2433985Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2434455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2434792Z res = mod(**inputs) 2025-11-03T16:37:32.2435149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2435505Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2435872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2436254Z outputs = self.model.decoder( 2025-11-03T16:37:32.2436669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2437058Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2437444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2437811Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2438164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2438535Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2438918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2439288Z return func(*args, **kwargs) 2025-11-03T16:37:32.2439642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2440044Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2440452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2440813Z return func(*args, **kwargs) 2025-11-03T16:37:32.2441179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2441560Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2441704Z 2025-11-03T16:37:32.2441806Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2442157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2442472Z res = mod(**inputs) 2025-11-03T16:37:32.2442787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2443128Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2443493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2443859Z outputs = self.model.decoder( 2025-11-03T16:37:32.2444195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2444552Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2444930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2445297Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2445645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2445997Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2446371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2446736Z return func(*args, **kwargs) 2025-11-03T16:37:32.2447114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2447496Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2447876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2448231Z return func(*args, **kwargs) 2025-11-03T16:37:32.2448578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2448968Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2449109Z 2025-11-03T16:37:32.2449197Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2449398Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2449629Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2449975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2450307Z res = mod(**inputs) 2025-11-03T16:37:32.2450619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2450963Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2451324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2451689Z outputs = self.model.decoder( 2025-11-03T16:37:32.2452026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2452358Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2452716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2453075Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2453416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2453760Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2454127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2454505Z return func(*args, **kwargs) 2025-11-03T16:37:32.2454852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2455232Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2455605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2455967Z return func(*args, **kwargs) 2025-11-03T16:37:32.2456315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2456701Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2457136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2457622Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2457810Z 2025-11-03T16:37:32.2457911Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2458259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2458571Z res = mod(**inputs) 2025-11-03T16:37:32.2458887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2459241Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2459610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2459975Z outputs = self.model.decoder( 2025-11-03T16:37:32.2460336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2460668Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2461030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2461388Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2461722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2462062Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2462426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2462782Z return func(*args, **kwargs) 2025-11-03T16:37:32.2463130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2463530Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2463908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2464266Z return func(*args, **kwargs) 2025-11-03T16:37:32.2464610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2464981Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2465113Z 2025-11-03T16:37:32.2465220Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2465557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2465866Z res = mod(**inputs) 2025-11-03T16:37:32.2466177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2466518Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2466874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2467239Z outputs = self.model.decoder( 2025-11-03T16:37:32.2467602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2467952Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2468321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2468684Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2469031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2469389Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2469761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2470127Z return func(*args, **kwargs) 2025-11-03T16:37:32.2470479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2470877Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2471012Z 2025-11-03T16:37:32.2471120Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2471473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2471783Z res = mod(**inputs) 2025-11-03T16:37:32.2472107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2472457Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2472826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2473198Z outputs = self.model.decoder( 2025-11-03T16:37:32.2473576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2473928Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2474416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2474806Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2475160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2475541Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2475934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2476307Z return func(*args, **kwargs) 2025-11-03T16:37:32.2476687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2477080Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2477265Z 2025-11-03T16:37:32.2477374Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2477742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2478072Z res = mod(**inputs) 2025-11-03T16:37:32.2478398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2478749Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2479129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2479509Z outputs = self.model.decoder( 2025-11-03T16:37:32.2479854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2480203Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2480581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2480956Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2481310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2481700Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2482071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2482440Z return func(*args, **kwargs) 2025-11-03T16:37:32.2482797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2483175Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2483313Z 2025-11-03T16:37:32.2483424Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2483772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2484092Z res = mod(**inputs) 2025-11-03T16:37:32.2484420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2484778Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2485132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2485490Z outputs = self.model.decoder( 2025-11-03T16:37:32.2485819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2486155Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2486511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2486863Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2487201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2487566Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2487934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2488285Z return func(*args, **kwargs) 2025-11-03T16:37:32.2488639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2489028Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2489416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2489776Z return func(*args, **kwargs) 2025-11-03T16:37:32.2490116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2490516Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2490703Z 2025-11-03T16:37:32.2490809Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2491156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2491472Z res = mod(**inputs) 2025-11-03T16:37:32.2491787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2492130Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2492840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2493214Z outputs = self.model.decoder( 2025-11-03T16:37:32.2493555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2493900Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2494264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2494619Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2494961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2495333Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2495699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2496051Z return func(*args, **kwargs) 2025-11-03T16:37:32.2496402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2496800Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2497178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2497527Z return func(*args, **kwargs) 2025-11-03T16:37:32.2497863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2498224Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2498385Z 2025-11-03T16:37:32.2498481Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2498815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2499121Z res = mod(**inputs) 2025-11-03T16:37:32.2499421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2499752Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2500098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2500452Z outputs = self.model.decoder( 2025-11-03T16:37:32.2500773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2501125Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2501483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2501848Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2502190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2502522Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2502877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2503222Z return func(*args, **kwargs) 2025-11-03T16:37:32.2503558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2503922Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2504313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2504665Z return func(*args, **kwargs) 2025-11-03T16:37:32.2505009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2505372Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2505503Z 2025-11-03T16:37:32.2505580Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2505786Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2506007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2506342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2506635Z res = mod(**inputs) 2025-11-03T16:37:32.2506944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2507279Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2507634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2508006Z outputs = self.model.decoder( 2025-11-03T16:37:32.2508326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2508659Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2509010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2509361Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2509691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2510032Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2510389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2510738Z return func(*args, **kwargs) 2025-11-03T16:37:32.2511080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2511464Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2511838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2512187Z return func(*args, **kwargs) 2025-11-03T16:37:32.2512525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2512898Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2513502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2513974Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2514217Z 2025-11-03T16:37:32.2514765Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2515141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2515479Z res = mod(**inputs) 2025-11-03T16:37:32.2515786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2516132Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2516503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2516855Z outputs = self.model.decoder( 2025-11-03T16:37:32.2517181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2517523Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2517914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2518275Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2518617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2518961Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2519326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2519683Z return func(*args, **kwargs) 2025-11-03T16:37:32.2520030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2520405Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2520791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2521152Z return func(*args, **kwargs) 2025-11-03T16:37:32.2521501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2521900Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2522033Z 2025-11-03T16:37:32.2522133Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2522476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2522788Z res = mod(**inputs) 2025-11-03T16:37:32.2523101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2523443Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2523801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2524161Z outputs = self.model.decoder( 2025-11-03T16:37:32.2524501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2524845Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2525244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2525603Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2525940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2526288Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2526651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2527000Z return func(*args, **kwargs) 2025-11-03T16:37:32.2527345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2527710Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2527841Z 2025-11-03T16:37:32.2527967Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2528319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2528638Z res = mod(**inputs) 2025-11-03T16:37:32.2528970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2529309Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2529668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2530024Z outputs = self.model.decoder( 2025-11-03T16:37:32.2530357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2530693Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2531070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2531419Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2531739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2532080Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2532434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2532782Z return func(*args, **kwargs) 2025-11-03T16:37:32.2533111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2533482Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2533627Z 2025-11-03T16:37:32.2533722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2534054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2534357Z res = mod(**inputs) 2025-11-03T16:37:32.2534654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2535005Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2535350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2535703Z outputs = self.model.decoder( 2025-11-03T16:37:32.2536026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2536349Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2536696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2537045Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2537375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2537705Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2538057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2538424Z return func(*args, **kwargs) 2025-11-03T16:37:32.2538762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2539118Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2539246Z 2025-11-03T16:37:32.2539342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2539677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2539977Z res = mod(**inputs) 2025-11-03T16:37:32.2540279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2540605Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2540972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2541327Z outputs = self.model.decoder( 2025-11-03T16:37:32.2541648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2541982Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2542324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2542674Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2543003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2543345Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2543718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2544062Z return func(*args, **kwargs) 2025-11-03T16:37:32.2544402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2544781Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2545155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2545496Z return func(*args, **kwargs) 2025-11-03T16:37:32.2545837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2546226Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2546380Z 2025-11-03T16:37:32.2546485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2546828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2547131Z res = mod(**inputs) 2025-11-03T16:37:32.2547444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2547795Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2548145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2548496Z outputs = self.model.decoder( 2025-11-03T16:37:32.2548814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2549142Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2549489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2549839Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2550160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2550501Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2550857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2551225Z return func(*args, **kwargs) 2025-11-03T16:37:32.2551559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2551928Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2552302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2552650Z return func(*args, **kwargs) 2025-11-03T16:37:32.2552985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2553333Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2553467Z 2025-11-03T16:37:32.2553581Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2553929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2554336Z res = mod(**inputs) 2025-11-03T16:37:32.2554664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2555006Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2555383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2555753Z outputs = self.model.decoder( 2025-11-03T16:37:32.2556101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2556455Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2556854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2557229Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2557577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2557950Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2558322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2558696Z return func(*args, **kwargs) 2025-11-03T16:37:32.2559055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2559471Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2559911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2560277Z return func(*args, **kwargs) 2025-11-03T16:37:32.2560640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2561027Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2561188Z 2025-11-03T16:37:32.2561278Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2561484Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2561717Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2562073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2562392Z res = mod(**inputs) 2025-11-03T16:37:32.2562714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2563057Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2563428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2563800Z outputs = self.model.decoder( 2025-11-03T16:37:32.2564153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2564501Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2564852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2565206Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2565546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2565899Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2566258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2566615Z return func(*args, **kwargs) 2025-11-03T16:37:32.2566968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2567380Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2567759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2568107Z return func(*args, **kwargs) 2025-11-03T16:37:32.2568454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2568830Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2569249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2569702Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2569885Z 2025-11-03T16:37:32.2569985Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2570349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2570663Z res = mod(**inputs) 2025-11-03T16:37:32.2570986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2571316Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2571668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2572023Z outputs = self.model.decoder( 2025-11-03T16:37:32.2572353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2572693Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2573045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2573409Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2573740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2574081Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2574432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2574819Z return func(*args, **kwargs) 2025-11-03T16:37:32.2575158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2575532Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2575907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2576248Z return func(*args, **kwargs) 2025-11-03T16:37:32.2576588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2576951Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2577082Z 2025-11-03T16:37:32.2577187Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2577522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2577838Z res = mod(**inputs) 2025-11-03T16:37:32.2578148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2578481Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2578835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2579182Z outputs = self.model.decoder( 2025-11-03T16:37:32.2579521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2579854Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2580221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2580576Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2580905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2581256Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2581623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2581986Z return func(*args, **kwargs) 2025-11-03T16:37:32.2582328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2582698Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2582845Z 2025-11-03T16:37:32.2582948Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2583332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2583652Z res = mod(**inputs) 2025-11-03T16:37:32.2583974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2584320Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2584679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2585042Z outputs = self.model.decoder( 2025-11-03T16:37:32.2585373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2585704Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2586061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2586419Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2586759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2587102Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2587475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2587858Z return func(*args, **kwargs) 2025-11-03T16:37:32.2588213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2588603Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2588753Z 2025-11-03T16:37:32.2588856Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2589210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2589531Z res = mod(**inputs) 2025-11-03T16:37:32.2589852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2590197Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2590570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2590951Z outputs = self.model.decoder( 2025-11-03T16:37:32.2591287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2591627Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2591975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2592333Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2592733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2593084Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2593450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2593816Z return func(*args, **kwargs) 2025-11-03T16:37:32.2594235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2594622Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2594756Z 2025-11-03T16:37:32.2594864Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2595203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2595517Z res = mod(**inputs) 2025-11-03T16:37:32.2595835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2596183Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2596549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2596950Z outputs = self.model.decoder( 2025-11-03T16:37:32.2597290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2597622Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2597970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2598312Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2598637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2598975Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2599328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2599676Z return func(*args, **kwargs) 2025-11-03T16:37:32.2600008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-11-03T16:37:32.2600413Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-11-03T16:37:32.2600597Z 2025-11-03T16:37:32.2600698Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2601054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2601359Z res = mod(**inputs) 2025-11-03T16:37:32.2601655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2601987Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2602334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2602690Z outputs = self.model.decoder( 2025-11-03T16:37:32.2603002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2603334Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2603684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2604055Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2604387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2604725Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2605089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2605443Z return func(*args, **kwargs) 2025-11-03T16:37:32.2605786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2606164Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2606540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2606922Z return func(*args, **kwargs) 2025-11-03T16:37:32.2607264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2607666Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2607824Z 2025-11-03T16:37:32.2607925Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2608268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2608582Z res = mod(**inputs) 2025-11-03T16:37:32.2608898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2609239Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2609589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2609967Z outputs = self.model.decoder( 2025-11-03T16:37:32.2610292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2610626Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2610969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2611321Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2611646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2611982Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2612335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2612675Z return func(*args, **kwargs) 2025-11-03T16:37:32.2613014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2613521Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2613903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2614295Z return func(*args, **kwargs) 2025-11-03T16:37:32.2614627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2614987Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2615120Z 2025-11-03T16:37:32.2615217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2615550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2615844Z res = mod(**inputs) 2025-11-03T16:37:32.2616148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2616482Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2616836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2617223Z outputs = self.model.decoder( 2025-11-03T16:37:32.2617556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2617893Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2618244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2618600Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2618932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2619278Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2619641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2620021Z return func(*args, **kwargs) 2025-11-03T16:37:32.2620360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2620731Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2621105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2621451Z return func(*args, **kwargs) 2025-11-03T16:37:32.2621789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2622147Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2622278Z 2025-11-03T16:37:32.2622355Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2622559Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2622783Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2623167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2623467Z res = mod(**inputs) 2025-11-03T16:37:32.2623777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2624110Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2624463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2624819Z outputs = self.model.decoder( 2025-11-03T16:37:32.2625135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2625465Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2625814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2626167Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2626492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2626838Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2627218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2627567Z return func(*args, **kwargs) 2025-11-03T16:37:32.2627902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2628270Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2628645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2628993Z return func(*args, **kwargs) 2025-11-03T16:37:32.2629330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2629711Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2630121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2630592Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2630772Z 2025-11-03T16:37:32.2630871Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2631205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2631503Z res = mod(**inputs) 2025-11-03T16:37:32.2631799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2632132Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2632487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2632855Z outputs = self.model.decoder( 2025-11-03T16:37:32.2633175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2633508Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2633863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2634289Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2634633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2634984Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2635357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2635723Z return func(*args, **kwargs) 2025-11-03T16:37:32.2636104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2636490Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2636880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2637239Z return func(*args, **kwargs) 2025-11-03T16:37:32.2637586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2637952Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2638084Z 2025-11-03T16:37:32.2638182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2638527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2638835Z res = mod(**inputs) 2025-11-03T16:37:32.2639147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2639489Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2639841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2640235Z outputs = self.model.decoder( 2025-11-03T16:37:32.2640569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2640905Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2641255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2641615Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2641955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2642308Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2642680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2643029Z return func(*args, **kwargs) 2025-11-03T16:37:32.2643371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2643758Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2643889Z 2025-11-03T16:37:32.2643995Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2644333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2644642Z res = mod(**inputs) 2025-11-03T16:37:32.2644951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2645292Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2645651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2646005Z outputs = self.model.decoder( 2025-11-03T16:37:32.2646360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2646708Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2647068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2647431Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2647763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2648117Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2648483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2648844Z return func(*args, **kwargs) 2025-11-03T16:37:32.2649212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2649588Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2649740Z 2025-11-03T16:37:32.2649837Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2650177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2650483Z res = mod(**inputs) 2025-11-03T16:37:32.2650782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2651117Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2651465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2651816Z outputs = self.model.decoder( 2025-11-03T16:37:32.2652132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2652463Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2652813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2653213Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2653542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2653879Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2654236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2654588Z return func(*args, **kwargs) 2025-11-03T16:37:32.2654926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2655284Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2655412Z 2025-11-03T16:37:32.2655508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2655847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2656152Z res = mod(**inputs) 2025-11-03T16:37:32.2656488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2656828Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2657192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2657569Z outputs = self.model.decoder( 2025-11-03T16:37:32.2657904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2658248Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2658602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2658966Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2659322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2659668Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2660018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2660369Z return func(*args, **kwargs) 2025-11-03T16:37:32.2660708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2661086Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2661463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2661806Z return func(*args, **kwargs) 2025-11-03T16:37:32.2662165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2662555Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2662708Z 2025-11-03T16:37:32.2662816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2663153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2663449Z res = mod(**inputs) 2025-11-03T16:37:32.2663753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2664083Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2664443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2664790Z outputs = self.model.decoder( 2025-11-03T16:37:32.2665112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2665445Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2665796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2666164Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2666488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2666830Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2667183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2667529Z return func(*args, **kwargs) 2025-11-03T16:37:32.2667867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2668233Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2668604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2668954Z return func(*args, **kwargs) 2025-11-03T16:37:32.2669290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2669656Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2669791Z 2025-11-03T16:37:32.2669888Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2670222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2670523Z res = mod(**inputs) 2025-11-03T16:37:32.2670824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2671145Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2671497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2671850Z outputs = self.model.decoder( 2025-11-03T16:37:32.2672202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2672530Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2672892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2673242Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2673571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2673926Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2674368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2674744Z return func(*args, **kwargs) 2025-11-03T16:37:32.2675124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2675525Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2675925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2676290Z return func(*args, **kwargs) 2025-11-03T16:37:32.2676649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2677039Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2677178Z 2025-11-03T16:37:32.2677266Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2677471Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2677704Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2678062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2678379Z res = mod(**inputs) 2025-11-03T16:37:32.2678708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2679056Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2679448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2679820Z outputs = self.model.decoder( 2025-11-03T16:37:32.2680164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2680507Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2680875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2681242Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2681590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2681957Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2682325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2682714Z return func(*args, **kwargs) 2025-11-03T16:37:32.2683083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2683478Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2683874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2684271Z return func(*args, **kwargs) 2025-11-03T16:37:32.2684625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2685024Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2685464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2685947Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2686137Z 2025-11-03T16:37:32.2686241Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2686596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2686915Z res = mod(**inputs) 2025-11-03T16:37:32.2687238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2687581Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2687955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2688327Z outputs = self.model.decoder( 2025-11-03T16:37:32.2688660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2689009Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2689353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2689707Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2690034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2690371Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2690716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2691062Z return func(*args, **kwargs) 2025-11-03T16:37:32.2691399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2691770Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2692148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2692486Z return func(*args, **kwargs) 2025-11-03T16:37:32.2692824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2693198Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2693327Z 2025-11-03T16:37:32.2693431Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2693769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2694066Z res = mod(**inputs) 2025-11-03T16:37:32.2694370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2694702Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2695054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2695401Z outputs = self.model.decoder( 2025-11-03T16:37:32.2695725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2696071Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2696420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2696773Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2697092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2697434Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2697789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2698139Z return func(*args, **kwargs) 2025-11-03T16:37:32.2698468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2698844Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2698981Z 2025-11-03T16:37:32.2699078Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2699412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2699713Z res = mod(**inputs) 2025-11-03T16:37:32.2700009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2700344Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2700693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2701044Z outputs = self.model.decoder( 2025-11-03T16:37:32.2701363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2701711Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2702064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2702422Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2702757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2703106Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2703462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2703814Z return func(*args, **kwargs) 2025-11-03T16:37:32.2704158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2704543Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2704690Z 2025-11-03T16:37:32.2704791Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2705136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2705450Z res = mod(**inputs) 2025-11-03T16:37:32.2705776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2706112Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2706477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2706847Z outputs = self.model.decoder( 2025-11-03T16:37:32.2707189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2707520Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2707866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2708223Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2708566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2708934Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2709293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2709653Z return func(*args, **kwargs) 2025-11-03T16:37:32.2710000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2710369Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2710500Z 2025-11-03T16:37:32.2710606Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2710945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2711255Z res = mod(**inputs) 2025-11-03T16:37:32.2711587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2711929Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2712289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2712644Z outputs = self.model.decoder( 2025-11-03T16:37:32.2712974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2713471Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2713836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2714272Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2714624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2714988Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2715418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2715787Z return func(*args, **kwargs) 2025-11-03T16:37:32.2716143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-11-03T16:37:32.2716570Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-11-03T16:37:32.2716763Z 2025-11-03T16:37:32.2716867Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2717220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2717541Z res = mod(**inputs) 2025-11-03T16:37:32.2717855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2718212Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2718588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2718964Z outputs = self.model.decoder( 2025-11-03T16:37:32.2719312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2719650Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2720009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2720370Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2720705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2721049Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2721410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2721772Z return func(*args, **kwargs) 2025-11-03T16:37:32.2722121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2722551Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2722937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2723297Z return func(*args, **kwargs) 2025-11-03T16:37:32.2723645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2724049Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2724202Z 2025-11-03T16:37:32.2724300Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2724637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2724941Z res = mod(**inputs) 2025-11-03T16:37:32.2725291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2725625Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2725975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2726328Z outputs = self.model.decoder( 2025-11-03T16:37:32.2726653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2726980Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2727323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2727673Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2728004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2728366Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2728720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2729064Z return func(*args, **kwargs) 2025-11-03T16:37:32.2729403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2729783Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2730157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2730504Z return func(*args, **kwargs) 2025-11-03T16:37:32.2730834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2731188Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2731322Z 2025-11-03T16:37:32.2731418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2731755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2732052Z res = mod(**inputs) 2025-11-03T16:37:32.2732371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2732705Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2733060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2733414Z outputs = self.model.decoder( 2025-11-03T16:37:32.2733737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2734073Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2734425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2734773Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2735095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2735455Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2735813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2736167Z return func(*args, **kwargs) 2025-11-03T16:37:32.2736512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2736883Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2737261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2737615Z return func(*args, **kwargs) 2025-11-03T16:37:32.2737956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2738338Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2738473Z 2025-11-03T16:37:32.2738553Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2738757Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2738981Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2739317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2739613Z res = mod(**inputs) 2025-11-03T16:37:32.2739925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2740259Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2740612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2740971Z outputs = self.model.decoder( 2025-11-03T16:37:32.2741309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2741639Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2741994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2742349Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2742670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2743011Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2743370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2743720Z return func(*args, **kwargs) 2025-11-03T16:37:32.2744060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2744427Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2744804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2745168Z return func(*args, **kwargs) 2025-11-03T16:37:32.2745512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2745889Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2746297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2746748Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2746924Z 2025-11-03T16:37:32.2747023Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2747360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2747651Z res = mod(**inputs) 2025-11-03T16:37:32.2747957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2748312Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2748670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2749030Z outputs = self.model.decoder( 2025-11-03T16:37:32.2749352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2749687Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2750042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2750398Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2750732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2751074Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2751453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2751802Z return func(*args, **kwargs) 2025-11-03T16:37:32.2752141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2752506Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2752886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2753262Z return func(*args, **kwargs) 2025-11-03T16:37:32.2753609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2753979Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2754180Z 2025-11-03T16:37:32.2754308Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2754665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2754980Z res = mod(**inputs) 2025-11-03T16:37:32.2755307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2755652Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2756027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2756405Z outputs = self.model.decoder( 2025-11-03T16:37:32.2756785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2757130Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2757483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2757848Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2758185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2758555Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2758919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2759271Z return func(*args, **kwargs) 2025-11-03T16:37:32.2759620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2759991Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2760123Z 2025-11-03T16:37:32.2760232Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2760572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2760885Z res = mod(**inputs) 2025-11-03T16:37:32.2761205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2761552Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2761934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2762292Z outputs = self.model.decoder( 2025-11-03T16:37:32.2762628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2762966Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2763323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2763672Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2764013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2764367Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2764748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2765111Z return func(*args, **kwargs) 2025-11-03T16:37:32.2765451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2765835Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2765988Z 2025-11-03T16:37:32.2766088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2766430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2766746Z res = mod(**inputs) 2025-11-03T16:37:32.2767043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2767377Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2767753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2768109Z outputs = self.model.decoder( 2025-11-03T16:37:32.2768430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2768760Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2769108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2769458Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2769789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2770121Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2770474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2770822Z return func(*args, **kwargs) 2025-11-03T16:37:32.2771160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2771549Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2771676Z 2025-11-03T16:37:32.2771773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2772109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2772416Z res = mod(**inputs) 2025-11-03T16:37:32.2772731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2773056Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2773418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2773776Z outputs = self.model.decoder( 2025-11-03T16:37:32.2774109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2774449Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2774819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2775182Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2775532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2775875Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2776225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2776575Z return func(*args, **kwargs) 2025-11-03T16:37:32.2776918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2777299Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2777700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2778054Z return func(*args, **kwargs) 2025-11-03T16:37:32.2778290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2778402Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2778406Z 2025-11-03T16:37:32.2778513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2778703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2778766Z res = mod(**inputs) 2025-11-03T16:37:32.2778984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2779056Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2779308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2779381Z outputs = self.model.decoder( 2025-11-03T16:37:32.2779601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2779673Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2779908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2779986Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2780203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2780288Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2780521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2780589Z return func(*args, **kwargs) 2025-11-03T16:37:32.2780833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2780932Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2781183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2781250Z return func(*args, **kwargs) 2025-11-03T16:37:32.2781476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2781562Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2781566Z 2025-11-03T16:37:32.2781666Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2781861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2781923Z res = mod(**inputs) 2025-11-03T16:37:32.2782138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2782210Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2782455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2782534Z outputs = self.model.decoder( 2025-11-03T16:37:32.2782745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2782829Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2783072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2783144Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2783374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2783453Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2783715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2783785Z return func(*args, **kwargs) 2025-11-03T16:37:32.2784020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2784124Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2784359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2784435Z return func(*args, **kwargs) 2025-11-03T16:37:32.2784687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2784776Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2784780Z 2025-11-03T16:37:32.2784858Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2784953Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2785061Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2785251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2785322Z res = mod(**inputs) 2025-11-03T16:37:32.2785532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2785602Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2785841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2785911Z outputs = self.model.decoder( 2025-11-03T16:37:32.2786127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2786197Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2786431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2786506Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2786722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2786820Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2787048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2787122Z return func(*args, **kwargs) 2025-11-03T16:37:32.2787350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2787443Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2787682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2787749Z return func(*args, **kwargs) 2025-11-03T16:37:32.2787986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2788097Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2788380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2788518Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2788521Z 2025-11-03T16:37:32.2788620Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2788821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2788884Z res = mod(**inputs) 2025-11-03T16:37:32.2789102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2789173Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2789420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2789501Z outputs = self.model.decoder( 2025-11-03T16:37:32.2789710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2789788Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2790017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2790087Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2790306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2790382Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2790621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2790704Z return func(*args, **kwargs) 2025-11-03T16:37:32.2790939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2791045Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2791278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2791352Z return func(*args, **kwargs) 2025-11-03T16:37:32.2791585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2791672Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2791675Z 2025-11-03T16:37:32.2791773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2791962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2792032Z res = mod(**inputs) 2025-11-03T16:37:32.2792243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2792322Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2792567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2792639Z outputs = self.model.decoder( 2025-11-03T16:37:32.2792857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2792926Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2793164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2793234Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2793445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2793530Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2793761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2793852Z return func(*args, **kwargs) 2025-11-03T16:37:32.2794144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2794257Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2794261Z 2025-11-03T16:37:32.2794366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2794558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2794630Z res = mod(**inputs) 2025-11-03T16:37:32.2794840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2794921Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2795180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2795253Z outputs = self.model.decoder( 2025-11-03T16:37:32.2795473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2795548Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2795793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2795866Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2796081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2796164Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2796401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2796502Z return func(*args, **kwargs) 2025-11-03T16:37:32.2796739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2796845Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2796849Z 2025-11-03T16:37:32.2796955Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2797141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2797210Z res = mod(**inputs) 2025-11-03T16:37:32.2797414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2797490Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2797717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2797788Z outputs = self.model.decoder( 2025-11-03T16:37:32.2798005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2798073Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2798308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2798392Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2798601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2798681Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2798908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2798980Z return func(*args, **kwargs) 2025-11-03T16:37:32.2799209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2799294Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2799297Z 2025-11-03T16:37:32.2799393Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2799575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2799662Z res = mod(**inputs) 2025-11-03T16:37:32.2799864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2799940Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2800162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2800230Z outputs = self.model.decoder( 2025-11-03T16:37:32.2800440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2800508Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2800752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2800821Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2801027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2801110Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2801343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2801418Z return func(*args, **kwargs) 2025-11-03T16:37:32.2801641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-11-03T16:37:32.2801774Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-11-03T16:37:32.2801777Z 2025-11-03T16:37:32.2801873Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2802070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2802144Z res = mod(**inputs) 2025-11-03T16:37:32.2802348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2802427Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2802651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2802720Z outputs = self.model.decoder( 2025-11-03T16:37:32.2802931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2803000Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2803233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2803300Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2803519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2803594Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2803823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2803910Z return func(*args, **kwargs) 2025-11-03T16:37:32.2804141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2804256Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2804486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2804554Z return func(*args, **kwargs) 2025-11-03T16:37:32.2804788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2804896Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2804901Z 2025-11-03T16:37:32.2805004Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2805209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2805273Z res = mod(**inputs) 2025-11-03T16:37:32.2805484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2805553Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2805786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2805854Z outputs = self.model.decoder( 2025-11-03T16:37:32.2806061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2806131Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2806371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2806450Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2806660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2806741Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2806964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2807028Z return func(*args, **kwargs) 2025-11-03T16:37:32.2807256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2807347Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2807576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2807654Z return func(*args, **kwargs) 2025-11-03T16:37:32.2807881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2807967Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2807971Z 2025-11-03T16:37:32.2808066Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2808257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2808316Z res = mod(**inputs) 2025-11-03T16:37:32.2808523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2808591Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2808813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2808888Z outputs = self.model.decoder( 2025-11-03T16:37:32.2809093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2809171Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2809395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2809496Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2809715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2809788Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2810020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2810085Z return func(*args, **kwargs) 2025-11-03T16:37:32.2810314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2810413Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2810640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2810730Z return func(*args, **kwargs) 2025-11-03T16:37:32.2810956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2811043Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2811047Z 2025-11-03T16:37:32.2811121Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2811195Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2811299Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2811483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2811551Z res = mod(**inputs) 2025-11-03T16:37:32.2811761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2811846Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2812076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2812148Z outputs = self.model.decoder( 2025-11-03T16:37:32.2812356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2812426Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2812647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2812721Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2812924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2813005Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2813406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2813489Z return func(*args, **kwargs) 2025-11-03T16:37:32.2813720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2813817Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2814056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2814124Z return func(*args, **kwargs) 2025-11-03T16:37:32.2814359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2814455Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2814730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2814865Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2814869Z 2025-11-03T16:37:32.2814967Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2815188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2815250Z res = mod(**inputs) 2025-11-03T16:37:32.2815461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2815532Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2815755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2815834Z outputs = self.model.decoder( 2025-11-03T16:37:32.2816036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2816112Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2816341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2816409Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2816646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2816720Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2816957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2817023Z return func(*args, **kwargs) 2025-11-03T16:37:32.2817243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2817342Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2817563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2817637Z return func(*args, **kwargs) 2025-11-03T16:37:32.2817893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2817984Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2817988Z 2025-11-03T16:37:32.2818083Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2818266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2818335Z res = mod(**inputs) 2025-11-03T16:37:32.2818538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2818614Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2818837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2818907Z outputs = self.model.decoder( 2025-11-03T16:37:32.2819131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2819200Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2819436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2819505Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2819712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2819793Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2820018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2820091Z return func(*args, **kwargs) 2025-11-03T16:37:32.2820315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2820401Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2820405Z 2025-11-03T16:37:32.2820500Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2820685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2820772Z res = mod(**inputs) 2025-11-03T16:37:32.2820981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2821060Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2821284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2821354Z outputs = self.model.decoder( 2025-11-03T16:37:32.2821563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2821633Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2821869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2821936Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2822160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2822242Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2822467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2822539Z return func(*args, **kwargs) 2025-11-03T16:37:32.2822762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2822859Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2822862Z 2025-11-03T16:37:32.2822956Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2823138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2823220Z res = mod(**inputs) 2025-11-03T16:37:32.2823423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2823502Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2823726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2823795Z outputs = self.model.decoder( 2025-11-03T16:37:32.2824004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2824072Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2824303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2824369Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2824591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2824670Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2824894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2824969Z return func(*args, **kwargs) 2025-11-03T16:37:32.2825194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2825276Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2825280Z 2025-11-03T16:37:32.2825372Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2825556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2825624Z res = mod(**inputs) 2025-11-03T16:37:32.2825826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2825902Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2826126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2826209Z outputs = self.model.decoder( 2025-11-03T16:37:32.2826417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2826485Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2826713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2826779Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2826987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2827069Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2827304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2827377Z return func(*args, **kwargs) 2025-11-03T16:37:32.2827616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2827716Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2827943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2828007Z return func(*args, **kwargs) 2025-11-03T16:37:32.2828239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2828343Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2828346Z 2025-11-03T16:37:32.2828449Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2828634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2828709Z res = mod(**inputs) 2025-11-03T16:37:32.2828919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2828990Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2829217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2829288Z outputs = self.model.decoder( 2025-11-03T16:37:32.2829497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2829567Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2829790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2829868Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2830089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2830172Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2830396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2830462Z return func(*args, **kwargs) 2025-11-03T16:37:32.2830699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2830789Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2831020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2831085Z return func(*args, **kwargs) 2025-11-03T16:37:32.2831318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2831403Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2831407Z 2025-11-03T16:37:32.2831504Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2831696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2831772Z res = mod(**inputs) 2025-11-03T16:37:32.2831983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2832053Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2832275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2832351Z outputs = self.model.decoder( 2025-11-03T16:37:32.2832558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2832634Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2832858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2832927Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2833153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2833228Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2833459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2833524Z return func(*args, **kwargs) 2025-11-03T16:37:32.2833747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2833845Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2834137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2834222Z return func(*args, **kwargs) 2025-11-03T16:37:32.2834481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2834577Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2834583Z 2025-11-03T16:37:32.2834664Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2834743Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2834852Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2835052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2835126Z res = mod(**inputs) 2025-11-03T16:37:32.2835345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2835417Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2835667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2835755Z outputs = self.model.decoder( 2025-11-03T16:37:32.2836008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2836088Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2836328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2836409Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2836630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2836716Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2836954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2837029Z return func(*args, **kwargs) 2025-11-03T16:37:32.2837272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2837368Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2837615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2837699Z return func(*args, **kwargs) 2025-11-03T16:37:32.2837947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2838045Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2838339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2838475Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2838478Z 2025-11-03T16:37:32.2838578Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2838787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2838850Z res = mod(**inputs) 2025-11-03T16:37:32.2839080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2839179Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2839414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2839494Z outputs = self.model.decoder( 2025-11-03T16:37:32.2839707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2839786Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2840022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2840092Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2840331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2840411Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2840662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2840731Z return func(*args, **kwargs) 2025-11-03T16:37:32.2840968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2841073Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2841309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2841386Z return func(*args, **kwargs) 2025-11-03T16:37:32.2841625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2841728Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2841733Z 2025-11-03T16:37:32.2841834Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2842034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2842108Z res = mod(**inputs) 2025-11-03T16:37:32.2842327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2842406Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2842646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2842720Z outputs = self.model.decoder( 2025-11-03T16:37:32.2842942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2843013Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2843262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2843332Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2843570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2843657Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2843894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2843971Z return func(*args, **kwargs) 2025-11-03T16:37:32.2844206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2844292Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2844296Z 2025-11-03T16:37:32.2844397Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2844588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2844662Z res = mod(**inputs) 2025-11-03T16:37:32.2844880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2844979Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2845217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2845287Z outputs = self.model.decoder( 2025-11-03T16:37:32.2845494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2845562Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2845793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2845859Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2846078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2846158Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2846387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2846460Z return func(*args, **kwargs) 2025-11-03T16:37:32.2846684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2846780Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2846784Z 2025-11-03T16:37:32.2846878Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2847063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2847130Z res = mod(**inputs) 2025-11-03T16:37:32.2847357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2847435Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2847659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2847731Z outputs = self.model.decoder( 2025-11-03T16:37:32.2847943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2848012Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2848245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2848312Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2848526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2848609Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2848837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2848910Z return func(*args, **kwargs) 2025-11-03T16:37:32.2849138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2849240Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2849243Z 2025-11-03T16:37:32.2849338Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2849518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2849585Z res = mod(**inputs) 2025-11-03T16:37:32.2849786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2849861Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2850083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2850153Z outputs = self.model.decoder( 2025-11-03T16:37:32.2850360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2850446Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2850677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2850742Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2850951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2851029Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2851253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2851325Z return func(*args, **kwargs) 2025-11-03T16:37:32.2851548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-11-03T16:37:32.2851695Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-11-03T16:37:32.2851700Z 2025-11-03T16:37:32.2851797Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2851980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2852048Z res = mod(**inputs) 2025-11-03T16:37:32.2852249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2852324Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2852554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2852621Z outputs = self.model.decoder( 2025-11-03T16:37:32.2852837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2852923Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2853159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2853230Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2853445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2853519Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2853744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2853815Z return func(*args, **kwargs) 2025-11-03T16:37:32.2854038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2854138Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2854362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2854426Z return func(*args, **kwargs) 2025-11-03T16:37:32.2854657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2854784Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2854788Z 2025-11-03T16:37:32.2854891Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2855074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2855136Z res = mod(**inputs) 2025-11-03T16:37:32.2855345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2855413Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2855645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2855717Z outputs = self.model.decoder( 2025-11-03T16:37:32.2855925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2856012Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2856236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2856313Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2856520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2856605Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2856828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2856894Z return func(*args, **kwargs) 2025-11-03T16:37:32.2857144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2857237Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2857471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2857540Z return func(*args, **kwargs) 2025-11-03T16:37:32.2857773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2857860Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2857863Z 2025-11-03T16:37:32.2857960Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2858148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2858210Z res = mod(**inputs) 2025-11-03T16:37:32.2858434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2858506Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2858730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2858809Z outputs = self.model.decoder( 2025-11-03T16:37:32.2859010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2859083Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2859307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2859375Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2859591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2859664Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2859900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2859965Z return func(*args, **kwargs) 2025-11-03T16:37:32.2860190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2860317Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2860539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2860612Z return func(*args, **kwargs) 2025-11-03T16:37:32.2860835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2860921Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2860924Z 2025-11-03T16:37:32.2860998Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2861070Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2861175Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2861360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2861445Z res = mod(**inputs) 2025-11-03T16:37:32.2861647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2861715Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2861945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2862012Z outputs = self.model.decoder( 2025-11-03T16:37:32.2862223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2862291Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2862519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2862594Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2862816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2862901Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2863123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2863196Z return func(*args, **kwargs) 2025-11-03T16:37:32.2863418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2863507Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2863739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2863803Z return func(*args, **kwargs) 2025-11-03T16:37:32.2864048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2864140Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2864414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2864545Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2864549Z 2025-11-03T16:37:32.2864643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2864833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2864894Z res = mod(**inputs) 2025-11-03T16:37:32.2865106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2865175Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2865402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2865480Z outputs = self.model.decoder( 2025-11-03T16:37:32.2865683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2865775Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2865998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2866065Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2866280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2866355Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2866589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2866653Z return func(*args, **kwargs) 2025-11-03T16:37:32.2866879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2866976Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2867217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2867290Z return func(*args, **kwargs) 2025-11-03T16:37:32.2867514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2867598Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2867602Z 2025-11-03T16:37:32.2867695Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2867878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2867947Z res = mod(**inputs) 2025-11-03T16:37:32.2868150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2868241Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2868466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2868538Z outputs = self.model.decoder( 2025-11-03T16:37:32.2868748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2868817Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2869045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2869112Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2869319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2869400Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2869640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2869717Z return func(*args, **kwargs) 2025-11-03T16:37:32.2869947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2870031Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2870035Z 2025-11-03T16:37:32.2870132Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2870317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2870386Z res = mod(**inputs) 2025-11-03T16:37:32.2870589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2870667Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2870900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2870971Z outputs = self.model.decoder( 2025-11-03T16:37:32.2871182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2871267Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2871501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2871570Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2871775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2871856Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2872080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2872152Z return func(*args, **kwargs) 2025-11-03T16:37:32.2872381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2872479Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2872496Z 2025-11-03T16:37:32.2872591Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2872775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2872845Z res = mod(**inputs) 2025-11-03T16:37:32.2873049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2873125Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2873356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2873428Z outputs = self.model.decoder( 2025-11-03T16:37:32.2873652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2873737Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2873982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2874121Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2874353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2874441Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2874682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2874761Z return func(*args, **kwargs) 2025-11-03T16:37:32.2875000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2875088Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2875091Z 2025-11-03T16:37:32.2875214Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2875411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2875488Z res = mod(**inputs) 2025-11-03T16:37:32.2875704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2875787Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2876027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2876100Z outputs = self.model.decoder( 2025-11-03T16:37:32.2876322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2876396Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2876642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2876715Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2876941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2877048Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2877287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2877365Z return func(*args, **kwargs) 2025-11-03T16:37:32.2877602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2877704Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2877944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2878013Z return func(*args, **kwargs) 2025-11-03T16:37:32.2878260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2878371Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2878392Z 2025-11-03T16:37:32.2878499Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2878692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2878755Z res = mod(**inputs) 2025-11-03T16:37:32.2878977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2879049Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2879295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2879369Z outputs = self.model.decoder( 2025-11-03T16:37:32.2879590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2879682Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2879919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2880000Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2880218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2880301Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2880537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2880605Z return func(*args, **kwargs) 2025-11-03T16:37:32.2880848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2880944Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2881215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2881284Z return func(*args, **kwargs) 2025-11-03T16:37:32.2881535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2881622Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2881626Z 2025-11-03T16:37:32.2881726Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2881929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2881994Z res = mod(**inputs) 2025-11-03T16:37:32.2882217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2882291Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2882536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2882619Z outputs = self.model.decoder( 2025-11-03T16:37:32.2882837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2882934Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2883171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2883244Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2883475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2883556Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2883803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2883874Z return func(*args, **kwargs) 2025-11-03T16:37:32.2884112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2884217Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2884480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2884560Z return func(*args, **kwargs) 2025-11-03T16:37:32.2884795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2884885Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2884889Z 2025-11-03T16:37:32.2884969Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2885046Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2885155Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2885348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2885419Z res = mod(**inputs) 2025-11-03T16:37:32.2885652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2885726Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2885973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2886044Z outputs = self.model.decoder( 2025-11-03T16:37:32.2886265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2886336Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2886574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2886653Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2886875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2886979Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2887219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2887296Z return func(*args, **kwargs) 2025-11-03T16:37:32.2887535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2887631Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2887874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2887943Z return func(*args, **kwargs) 2025-11-03T16:37:32.2888188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2888286Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2888577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2888715Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2888736Z 2025-11-03T16:37:32.2888836Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2889036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2889101Z res = mod(**inputs) 2025-11-03T16:37:32.2889321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2889393Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2889630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2889705Z outputs = self.model.decoder( 2025-11-03T16:37:32.2889908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2889984Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2890209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2890293Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2890507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2890582Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2890811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2890876Z return func(*args, **kwargs) 2025-11-03T16:37:32.2891099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2891197Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2891433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2891507Z return func(*args, **kwargs) 2025-11-03T16:37:32.2891730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2891813Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2891816Z 2025-11-03T16:37:32.2891910Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2892093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2892159Z res = mod(**inputs) 2025-11-03T16:37:32.2892361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2892437Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2892678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2892750Z outputs = self.model.decoder( 2025-11-03T16:37:32.2892958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2893029Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2893261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2893327Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2893535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2893616Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2893838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2893908Z return func(*args, **kwargs) 2025-11-03T16:37:32.2894133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2894216Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2894235Z 2025-11-03T16:37:32.2894331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2894513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2894582Z res = mod(**inputs) 2025-11-03T16:37:32.2894786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2894863Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2895085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2895152Z outputs = self.model.decoder( 2025-11-03T16:37:32.2895361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2895431Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2895663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2895746Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2895958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2896040Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2896267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2896340Z return func(*args, **kwargs) 2025-11-03T16:37:32.2896580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2896677Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2896680Z 2025-11-03T16:37:32.2896779Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2896980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2897054Z res = mod(**inputs) 2025-11-03T16:37:32.2897259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2897335Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2897561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2897630Z outputs = self.model.decoder( 2025-11-03T16:37:32.2897847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2897918Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2898150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2898234Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2898441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2898525Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2898746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2898820Z return func(*args, **kwargs) 2025-11-03T16:37:32.2899041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2899125Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2899128Z 2025-11-03T16:37:32.2899224Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2899408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2899476Z res = mod(**inputs) 2025-11-03T16:37:32.2899681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2899758Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2900001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2900069Z outputs = self.model.decoder( 2025-11-03T16:37:32.2900281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2900353Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2900589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2900657Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2900866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2900949Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2901177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2901263Z return func(*args, **kwargs) 2025-11-03T16:37:32.2901489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 294, in forward 2025-11-03T16:37:32.2901622Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-11-03T16:37:32.2901626Z 2025-11-03T16:37:32.2901721Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2901905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2901975Z res = mod(**inputs) 2025-11-03T16:37:32.2902177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2902253Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2902534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2902604Z outputs = self.model.decoder( 2025-11-03T16:37:32.2902816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2902888Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2903128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2903197Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2903411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2903485Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2903710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2903798Z return func(*args, **kwargs) 2025-11-03T16:37:32.2904024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2904126Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2904350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2904415Z return func(*args, **kwargs) 2025-11-03T16:37:32.2904645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 161, in forward 2025-11-03T16:37:32.2904749Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:37:32.2904754Z 2025-11-03T16:37:32.2904855Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2905036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2905098Z res = mod(**inputs) 2025-11-03T16:37:32.2905309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2905381Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2905627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2905695Z outputs = self.model.decoder( 2025-11-03T16:37:32.2905904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2905974Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2906195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2906270Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2906475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2906559Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2906782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2906865Z return func(*args, **kwargs) 2025-11-03T16:37:32.2907103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2907196Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2907444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2907508Z return func(*args, **kwargs) 2025-11-03T16:37:32.2907737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-11-03T16:37:32.2907819Z key_states = self.k_proj(hidden_states) 2025-11-03T16:37:32.2907822Z 2025-11-03T16:37:32.2907919Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2908130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2908195Z res = mod(**inputs) 2025-11-03T16:37:32.2908407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2908475Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2908699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2908777Z outputs = self.model.decoder( 2025-11-03T16:37:32.2908981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2909057Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2909282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2909366Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2909584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2909660Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2909892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2909957Z return func(*args, **kwargs) 2025-11-03T16:37:32.2910180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2910280Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2910501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2910574Z return func(*args, **kwargs) 2025-11-03T16:37:32.2910799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 165, in forward 2025-11-03T16:37:32.2910887Z value_states = self.v_proj(hidden_states) 2025-11-03T16:37:32.2910892Z 2025-11-03T16:37:32.2910983Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2911067Z cudagraph partition due to non gpu ops 2025-11-03T16:37:32.2911174Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2911359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2911427Z res = mod(**inputs) 2025-11-03T16:37:32.2911632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2911702Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2911937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2912004Z outputs = self.model.decoder( 2025-11-03T16:37:32.2912246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2912315Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2912557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2912632Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2912836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2912918Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2913145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2913332Z return func(*args, **kwargs) 2025-11-03T16:37:32.2913563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2913656Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2913929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2913998Z return func(*args, **kwargs) 2025-11-03T16:37:32.2914285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 186, in forward 2025-11-03T16:37:32.2914389Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:32.2914684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:32.2914949Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:32.2914953Z 2025-11-03T16:37:32.2915054Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2915253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2915357Z res = mod(**inputs) 2025-11-03T16:37:32.2915575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2915648Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2915873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2915951Z outputs = self.model.decoder( 2025-11-03T16:37:32.2916156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2916234Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2916460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2916529Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2916745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2916822Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2917058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2917150Z return func(*args, **kwargs) 2025-11-03T16:37:32.2917376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 262, in forward 2025-11-03T16:37:32.2917474Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:32.2917701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2917772Z return func(*args, **kwargs) 2025-11-03T16:37:32.2918000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 198, in forward 2025-11-03T16:37:32.2918083Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:32.2918086Z 2025-11-03T16:37:32.2918185Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2918371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2948354Z res = mod(**inputs) 2025-11-03T16:37:32.2948759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2948842Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2949104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2949186Z outputs = self.model.decoder( 2025-11-03T16:37:32.2949404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2949487Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2949745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2949896Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2950115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2950215Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2950448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2950518Z return func(*args, **kwargs) 2025-11-03T16:37:32.2950757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-11-03T16:37:32.2950838Z hidden_states = self.fc1(hidden_states) 2025-11-03T16:37:32.2950844Z 2025-11-03T16:37:32.2950961Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2951160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2951267Z res = mod(**inputs) 2025-11-03T16:37:32.2951496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2951575Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2951823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2951901Z outputs = self.model.decoder( 2025-11-03T16:37:32.2952135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2952225Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2952453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2952533Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2952740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2952830Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2953060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2953154Z return func(*args, **kwargs) 2025-11-03T16:37:32.2953394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 289, in forward 2025-11-03T16:37:32.2953494Z hidden_states = self.activation_fn(hidden_states) 2025-11-03T16:37:32.2953498Z 2025-11-03T16:37:32.2953610Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2953808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2953874Z res = mod(**inputs) 2025-11-03T16:37:32.2954185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2954265Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2954517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-11-03T16:37:32.2954626Z outputs = self.model.decoder( 2025-11-03T16:37:32.2954850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2954933Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2955179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 656, in forward 2025-11-03T16:37:32.2955261Z layer_outputs = decoder_layer( 2025-11-03T16:37:32.2955490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:32.2955586Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:32.2955822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:32.2955891Z return func(*args, **kwargs) 2025-11-03T16:37:32.2956185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-11-03T16:37:32.2956274Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:32.2956278Z 2025-11-03T16:37:32.2956391Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2956588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2956653Z res = mod(**inputs) 2025-11-03T16:37:32.2956877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2956949Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2957193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 841, in forward 2025-11-03T16:37:32.2957288Z logits = self.lm_head(outputs[0]).contiguous() 2025-11-03T16:37:32.2957314Z 2025-11-03T16:37:32.2957422Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:32.2957620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:32.2957687Z res = mod(**inputs) 2025-11-03T16:37:32.2957908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper 2025-11-03T16:37:32.2957979Z output = func(self, *args, **kwargs) 2025-11-03T16:37:32.2958222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 847, in forward 2025-11-03T16:37:32.2958295Z loss = self.loss_function( 2025-11-03T16:37:32.2958535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-11-03T16:37:32.2958715Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-11-03T16:37:32.2958965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-11-03T16:37:32.2959165Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-11-03T16:37:32.2959187Z 2025-11-03T16:37:42.7657461Z Compilation time (from dynamo_timed): 15.600962184 2025-11-03T16:37:42.8399787Z pass 2025-11-03T16:37:42.8404235Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:42.8408989Z TIMING: _recursive_pre_grad_passes:0.007 _recursive_joint_graph_passes:0.34807 _recursive_post_grad_passes:0.07246 async_compile.wait:0.78646 code_gen:9.1442 inductor_compile:10.57667 backend_compile:13.17749 gc:0.00087 entire_frame_compile:15.60096 total_wall_time:15.60096 2025-11-03T16:37:42.8410490Z STATS: call_* op count: 379 | FakeTensorMode.__torch_dispatch__:6922 | FakeTensor.__torch_dispatch__:4035 | ProxyTorchDispatchMode.__torch_dispatch__:1932 2025-11-03T16:37:42.8411000Z Dynamo produced 1 graphs covering 379 ops with 0 graph breaks (0 unique) 2025-11-03T16:37:45.1485187Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:37:45.1486331Z import pynvml # type: ignore[import] 2025-11-03T16:37:48.2715048Z 2025-11-03T16:37:49.2744075Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:37:49.2744458Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:37:49.2753641Z cpu eval PLBartForCausalLM 2025-11-03T16:37:49.9546966Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:50.2342538Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:50.5471933Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:37:54.5959520Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5961422Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5961837Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5962298Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5962541Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5962783Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5963103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.5963617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.5963944Z res = mod(**inputs) 2025-11-03T16:37:54.5964384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.5964788Z outputs = self.model.decoder( 2025-11-03T16:37:54.5965499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.5965887Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.5966241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.5966608Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.5966986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5967355Z return func(*args, **kwargs) 2025-11-03T16:37:54.5967734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.5968137Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.5968529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5968892Z return func(*args, **kwargs) 2025-11-03T16:37:54.5969261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-11-03T16:37:54.5969780Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:37:54.5969983Z 2025-11-03T16:37:54.5970089Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.5970441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.5970753Z res = mod(**inputs) 2025-11-03T16:37:54.5971116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.5971500Z outputs = self.model.decoder( 2025-11-03T16:37:54.5971887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.5972280Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.5972639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.5973064Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.5973433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5973803Z return func(*args, **kwargs) 2025-11-03T16:37:54.5974180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.5974597Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.5974999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5975353Z return func(*args, **kwargs) 2025-11-03T16:37:54.5975733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-11-03T16:37:54.5976175Z key_states = self.k_proj(current_states) 2025-11-03T16:37:54.5976317Z 2025-11-03T16:37:54.5976432Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.5976780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.5977100Z res = mod(**inputs) 2025-11-03T16:37:54.5977465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.5977857Z outputs = self.model.decoder( 2025-11-03T16:37:54.5978254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.5978646Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.5978998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.5979383Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.5979762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5980127Z return func(*args, **kwargs) 2025-11-03T16:37:54.5980491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.5980897Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.5981401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5981769Z return func(*args, **kwargs) 2025-11-03T16:37:54.5982141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-11-03T16:37:54.5982536Z value_states = self.v_proj(current_states) 2025-11-03T16:37:54.5982686Z 2025-11-03T16:37:54.5982780Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.5983046Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.5983440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.5983810Z res = mod(**inputs) 2025-11-03T16:37:54.5984188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.5984566Z outputs = self.model.decoder( 2025-11-03T16:37:54.5984935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.5985342Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.5985717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.5986107Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.5986515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5986915Z return func(*args, **kwargs) 2025-11-03T16:37:54.5987336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.5987786Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.5988211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5988608Z return func(*args, **kwargs) 2025-11-03T16:37:54.5989014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-11-03T16:37:54.5989455Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:54.5989932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:54.5990474Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:54.5990679Z 2025-11-03T16:37:54.5990813Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.5991200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.5991533Z res = mod(**inputs) 2025-11-03T16:37:54.5991930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.5992363Z outputs = self.model.decoder( 2025-11-03T16:37:54.5992780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.5993168Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.5993505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.5993865Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.5994394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5994766Z return func(*args, **kwargs) 2025-11-03T16:37:54.5995139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.5995541Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.5995939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.5996319Z return func(*args, **kwargs) 2025-11-03T16:37:54.5996695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-11-03T16:37:54.5997108Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:54.5997246Z 2025-11-03T16:37:54.5997352Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.5997712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.5998030Z res = mod(**inputs) 2025-11-03T16:37:54.5998447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.5998842Z outputs = self.model.decoder( 2025-11-03T16:37:54.5999216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.5999608Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.5999952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6000287Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6000637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6000986Z return func(*args, **kwargs) 2025-11-03T16:37:54.6001343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6001775Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6001940Z 2025-11-03T16:37:54.6002042Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6002365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6002661Z res = mod(**inputs) 2025-11-03T16:37:54.6003003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6003369Z outputs = self.model.decoder( 2025-11-03T16:37:54.6003720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6004091Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6004432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6004780Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6005141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6005488Z return func(*args, **kwargs) 2025-11-03T16:37:54.6005847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6006265Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6006637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:37:54.6006962Z return self.act(input) 2025-11-03T16:37:54.6007069Z 2025-11-03T16:37:54.6007167Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6007523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6007827Z res = mod(**inputs) 2025-11-03T16:37:54.6008170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6008533Z outputs = self.model.decoder( 2025-11-03T16:37:54.6008895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6009260Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6009590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6009928Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6010278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6010631Z return func(*args, **kwargs) 2025-11-03T16:37:54.6010983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-11-03T16:37:54.6011424Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:54.6011553Z 2025-11-03T16:37:54.6011656Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6011986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6012283Z res = mod(**inputs) 2025-11-03T16:37:54.6012629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6013003Z outputs = self.model.decoder( 2025-11-03T16:37:54.6013544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6013924Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6014271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6014618Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6015024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6015365Z return func(*args, **kwargs) 2025-11-03T16:37:54.6015721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6016116Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6016493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6016841Z return func(*args, **kwargs) 2025-11-03T16:37:54.6017189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-11-03T16:37:54.6017661Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:37:54.6017862Z 2025-11-03T16:37:54.6017960Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6018295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6018585Z res = mod(**inputs) 2025-11-03T16:37:54.6018924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6019289Z outputs = self.model.decoder( 2025-11-03T16:37:54.6019649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6020011Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6020329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6020696Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6021056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6021409Z return func(*args, **kwargs) 2025-11-03T16:37:54.6021767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6022148Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6022521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6022871Z return func(*args, **kwargs) 2025-11-03T16:37:54.6023224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-11-03T16:37:54.6023590Z key_states = self.k_proj(current_states) 2025-11-03T16:37:54.6023726Z 2025-11-03T16:37:54.6023826Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6024164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6024496Z res = mod(**inputs) 2025-11-03T16:37:54.6024840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6025199Z outputs = self.model.decoder( 2025-11-03T16:37:54.6025565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6025931Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6026259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6026597Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6026947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6027294Z return func(*args, **kwargs) 2025-11-03T16:37:54.6027650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6028061Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6028426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6028773Z return func(*args, **kwargs) 2025-11-03T16:37:54.6029125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-11-03T16:37:54.6029506Z value_states = self.v_proj(current_states) 2025-11-03T16:37:54.6029638Z 2025-11-03T16:37:54.6029722Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.6029940Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6030274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6030591Z res = mod(**inputs) 2025-11-03T16:37:54.6030936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6031310Z outputs = self.model.decoder( 2025-11-03T16:37:54.6031676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6032049Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6032383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6032733Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6033092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6033453Z return func(*args, **kwargs) 2025-11-03T16:37:54.6033842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6034354Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6034756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6035117Z return func(*args, **kwargs) 2025-11-03T16:37:54.6035506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-11-03T16:37:54.6035910Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:54.6036347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:54.6036810Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:54.6036985Z 2025-11-03T16:37:54.6037090Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6037441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6037753Z res = mod(**inputs) 2025-11-03T16:37:54.6038133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6038507Z outputs = self.model.decoder( 2025-11-03T16:37:54.6038883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6039258Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6039595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6039947Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6040310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6040671Z return func(*args, **kwargs) 2025-11-03T16:37:54.6041040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6041462Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6041845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6042197Z return func(*args, **kwargs) 2025-11-03T16:37:54.6042558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-11-03T16:37:54.6042942Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:54.6043074Z 2025-11-03T16:37:54.6043178Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6043518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6043828Z res = mod(**inputs) 2025-11-03T16:37:54.6044203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6044584Z outputs = self.model.decoder( 2025-11-03T16:37:54.6044957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6045325Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6045664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6046015Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6046380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6046737Z return func(*args, **kwargs) 2025-11-03T16:37:54.6047125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6047548Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6047723Z 2025-11-03T16:37:54.6047827Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6048177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6048476Z res = mod(**inputs) 2025-11-03T16:37:54.6048824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6049195Z outputs = self.model.decoder( 2025-11-03T16:37:54.6049560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6049928Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6050255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6050659Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6051049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6051417Z return func(*args, **kwargs) 2025-11-03T16:37:54.6051776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6052184Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6052547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:37:54.6052873Z return self.act(input) 2025-11-03T16:37:54.6052986Z 2025-11-03T16:37:54.6053083Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6053423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6053722Z res = mod(**inputs) 2025-11-03T16:37:54.6054062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6054457Z outputs = self.model.decoder( 2025-11-03T16:37:54.6054820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6055184Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6055511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6056467Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6056950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6057378Z return func(*args, **kwargs) 2025-11-03T16:37:54.6057741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-11-03T16:37:54.6058146Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:54.6058279Z 2025-11-03T16:37:54.6058377Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6058716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6059019Z res = mod(**inputs) 2025-11-03T16:37:54.6059363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6059730Z outputs = self.model.decoder( 2025-11-03T16:37:54.6060081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6060443Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6060770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6061109Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6061498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6061846Z return func(*args, **kwargs) 2025-11-03T16:37:54.6062205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6062599Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6063034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6063453Z return func(*args, **kwargs) 2025-11-03T16:37:54.6063815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-11-03T16:37:54.6064253Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:37:54.6064443Z 2025-11-03T16:37:54.6064552Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6064887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6065183Z res = mod(**inputs) 2025-11-03T16:37:54.6065546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6065911Z outputs = self.model.decoder( 2025-11-03T16:37:54.6066272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6066636Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6066959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6067297Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6067655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6068002Z return func(*args, **kwargs) 2025-11-03T16:37:54.6068352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6069694Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6070067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6070413Z return func(*args, **kwargs) 2025-11-03T16:37:54.6070771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-11-03T16:37:54.6071199Z key_states = self.k_proj(current_states) 2025-11-03T16:37:54.6071339Z 2025-11-03T16:37:54.6071440Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6071782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6072095Z res = mod(**inputs) 2025-11-03T16:37:54.6072471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6072849Z outputs = self.model.decoder( 2025-11-03T16:37:54.6073225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6073600Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6073935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6074358Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6074744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6075119Z return func(*args, **kwargs) 2025-11-03T16:37:54.6075526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6075949Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6076339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6076712Z return func(*args, **kwargs) 2025-11-03T16:37:54.6077089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-11-03T16:37:54.6077496Z value_states = self.v_proj(current_states) 2025-11-03T16:37:54.6077637Z 2025-11-03T16:37:54.6077727Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.6077958Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6078313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6078628Z res = mod(**inputs) 2025-11-03T16:37:54.6078997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6079379Z outputs = self.model.decoder( 2025-11-03T16:37:54.6079772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6080172Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6080518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6080875Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6081244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6081608Z return func(*args, **kwargs) 2025-11-03T16:37:54.6081978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6082389Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6082784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6083166Z return func(*args, **kwargs) 2025-11-03T16:37:54.6083534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-11-03T16:37:54.6083948Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:54.6084386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:54.6084855Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:54.6085038Z 2025-11-03T16:37:54.6085138Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6085480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6085786Z res = mod(**inputs) 2025-11-03T16:37:54.6086158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6086534Z outputs = self.model.decoder( 2025-11-03T16:37:54.6086908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6087286Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6087619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6087967Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6088327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6088685Z return func(*args, **kwargs) 2025-11-03T16:37:54.6089135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6089531Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6089895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6090249Z return func(*args, **kwargs) 2025-11-03T16:37:54.6090604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-11-03T16:37:54.6090978Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:54.6091107Z 2025-11-03T16:37:54.6091211Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6091538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6091837Z res = mod(**inputs) 2025-11-03T16:37:54.6092181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6092548Z outputs = self.model.decoder( 2025-11-03T16:37:54.6092914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6093290Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6093616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6093955Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6094314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6094652Z return func(*args, **kwargs) 2025-11-03T16:37:54.6095008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6095420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6095579Z 2025-11-03T16:37:54.6095685Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6096019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6096331Z res = mod(**inputs) 2025-11-03T16:37:54.6096680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6097046Z outputs = self.model.decoder( 2025-11-03T16:37:54.6097408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6097775Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6098097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6098440Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6098798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6099160Z return func(*args, **kwargs) 2025-11-03T16:37:54.6099510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6099924Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6100290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:37:54.6100615Z return self.act(input) 2025-11-03T16:37:54.6100718Z 2025-11-03T16:37:54.6100822Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6101149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6101450Z res = mod(**inputs) 2025-11-03T16:37:54.6101802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6102199Z outputs = self.model.decoder( 2025-11-03T16:37:54.6102564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6102946Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6103279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6103626Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6103990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6104342Z return func(*args, **kwargs) 2025-11-03T16:37:54.6104710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-11-03T16:37:54.6105097Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:54.6105231Z 2025-11-03T16:37:54.6105337Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6105682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6105981Z res = mod(**inputs) 2025-11-03T16:37:54.6106348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6106719Z outputs = self.model.decoder( 2025-11-03T16:37:54.6107092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6107453Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6107782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6108121Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6108476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6108824Z return func(*args, **kwargs) 2025-11-03T16:37:54.6109175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6109586Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6109960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6110360Z return func(*args, **kwargs) 2025-11-03T16:37:54.6110724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-11-03T16:37:54.6111167Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:37:54.6111371Z 2025-11-03T16:37:54.6111471Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6111812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6112122Z res = mod(**inputs) 2025-11-03T16:37:54.6112500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6112881Z outputs = self.model.decoder( 2025-11-03T16:37:54.6113400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6113790Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6114187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6114548Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6114930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6115301Z return func(*args, **kwargs) 2025-11-03T16:37:54.6115718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6116128Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6116511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6116861Z return func(*args, **kwargs) 2025-11-03T16:37:54.6117216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-11-03T16:37:54.6117596Z key_states = self.k_proj(current_states) 2025-11-03T16:37:54.6117724Z 2025-11-03T16:37:54.6117830Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6118160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6118467Z res = mod(**inputs) 2025-11-03T16:37:54.6118829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6119211Z outputs = self.model.decoder( 2025-11-03T16:37:54.6119582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6120000Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6120331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6120674Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6121035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6121377Z return func(*args, **kwargs) 2025-11-03T16:37:54.6121752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6122162Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6122541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6122879Z return func(*args, **kwargs) 2025-11-03T16:37:54.6123253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-11-03T16:37:54.6123632Z value_states = self.v_proj(current_states) 2025-11-03T16:37:54.6123766Z 2025-11-03T16:37:54.6123851Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.6124076Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6124400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6124695Z res = mod(**inputs) 2025-11-03T16:37:54.6125039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6125405Z outputs = self.model.decoder( 2025-11-03T16:37:54.6125802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6126181Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6126513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6126855Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6127210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6127554Z return func(*args, **kwargs) 2025-11-03T16:37:54.6127909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6128303Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6128675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6129038Z return func(*args, **kwargs) 2025-11-03T16:37:54.6129388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-11-03T16:37:54.6129779Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:54.6130196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:54.6130651Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:54.6130826Z 2025-11-03T16:37:54.6130930Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6131256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6131557Z res = mod(**inputs) 2025-11-03T16:37:54.6131901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6132270Z outputs = self.model.decoder( 2025-11-03T16:37:54.6132629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6133014Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6133342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6133681Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6134030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6134371Z return func(*args, **kwargs) 2025-11-03T16:37:54.6134726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6135115Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6135489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6135833Z return func(*args, **kwargs) 2025-11-03T16:37:54.6136177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-11-03T16:37:54.6136575Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:54.6136710Z 2025-11-03T16:37:54.6136806Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6137141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6137435Z res = mod(**inputs) 2025-11-03T16:37:54.6137781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6138147Z outputs = self.model.decoder( 2025-11-03T16:37:54.6138508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6138889Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6139217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6139561Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6139917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6140265Z return func(*args, **kwargs) 2025-11-03T16:37:54.6140616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6141017Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6141185Z 2025-11-03T16:37:54.6141282Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6141614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6141930Z res = mod(**inputs) 2025-11-03T16:37:54.6142271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6142641Z outputs = self.model.decoder( 2025-11-03T16:37:54.6143001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6143372Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6143772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6144109Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6144466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6144814Z return func(*args, **kwargs) 2025-11-03T16:37:54.6145259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6145669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6146021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:37:54.6146365Z return self.act(input) 2025-11-03T16:37:54.6146475Z 2025-11-03T16:37:54.6146571Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6146904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6147199Z res = mod(**inputs) 2025-11-03T16:37:54.6147542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6147918Z outputs = self.model.decoder( 2025-11-03T16:37:54.6148274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6148637Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6148954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6149312Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6149670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6150017Z return func(*args, **kwargs) 2025-11-03T16:37:54.6150371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-11-03T16:37:54.6150749Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:54.6150886Z 2025-11-03T16:37:54.6150985Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6151328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6151635Z res = mod(**inputs) 2025-11-03T16:37:54.6151990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6152370Z outputs = self.model.decoder( 2025-11-03T16:37:54.6152745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6153120Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6153454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6153796Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6154259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6154646Z return func(*args, **kwargs) 2025-11-03T16:37:54.6155040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6155492Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6155871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6156229Z return func(*args, **kwargs) 2025-11-03T16:37:54.6156592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-11-03T16:37:54.6157051Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:37:54.6157245Z 2025-11-03T16:37:54.6157345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6157683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6157985Z res = mod(**inputs) 2025-11-03T16:37:54.6158339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6158718Z outputs = self.model.decoder( 2025-11-03T16:37:54.6159083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6159482Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6159813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6160170Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6160543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6160893Z return func(*args, **kwargs) 2025-11-03T16:37:54.6161255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6161648Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6162028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6162379Z return func(*args, **kwargs) 2025-11-03T16:37:54.6162769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-11-03T16:37:54.6163153Z key_states = self.k_proj(current_states) 2025-11-03T16:37:54.6163281Z 2025-11-03T16:37:54.6163387Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6163724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6164026Z res = mod(**inputs) 2025-11-03T16:37:54.6164379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6164751Z outputs = self.model.decoder( 2025-11-03T16:37:54.6165118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6165513Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6165845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6166204Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6166558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6166905Z return func(*args, **kwargs) 2025-11-03T16:37:54.6167253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6167639Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6168008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6168359Z return func(*args, **kwargs) 2025-11-03T16:37:54.6168729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-11-03T16:37:54.6169119Z value_states = self.v_proj(current_states) 2025-11-03T16:37:54.6169267Z 2025-11-03T16:37:54.6169346Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.6169576Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6169918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6170220Z res = mod(**inputs) 2025-11-03T16:37:54.6170574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6170954Z outputs = self.model.decoder( 2025-11-03T16:37:54.6171327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6171706Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6172039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6172392Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6172775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6173134Z return func(*args, **kwargs) 2025-11-03T16:37:54.6173497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6173885Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6174270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6174630Z return func(*args, **kwargs) 2025-11-03T16:37:54.6174992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-11-03T16:37:54.6175387Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:54.6175815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:54.6176294Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:54.6176476Z 2025-11-03T16:37:54.6176574Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6176915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6177217Z res = mod(**inputs) 2025-11-03T16:37:54.6177568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6177946Z outputs = self.model.decoder( 2025-11-03T16:37:54.6178321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6178711Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6179050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6179399Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6179764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6180119Z return func(*args, **kwargs) 2025-11-03T16:37:54.6180476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6180875Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6181255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6181611Z return func(*args, **kwargs) 2025-11-03T16:37:54.6181990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-11-03T16:37:54.6182372Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:54.6182513Z 2025-11-03T16:37:54.6182615Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6182966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6183282Z res = mod(**inputs) 2025-11-03T16:37:54.6183643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6184030Z outputs = self.model.decoder( 2025-11-03T16:37:54.6184397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6184772Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6185110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6185448Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6185811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6186194Z return func(*args, **kwargs) 2025-11-03T16:37:54.6186571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6186989Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6187154Z 2025-11-03T16:37:54.6187252Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6187592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6187897Z res = mod(**inputs) 2025-11-03T16:37:54.6188245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6188623Z outputs = self.model.decoder( 2025-11-03T16:37:54.6188986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6189382Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6189716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6190060Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6190418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6190774Z return func(*args, **kwargs) 2025-11-03T16:37:54.6191140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6191554Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6191944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:37:54.6192269Z return self.act(input) 2025-11-03T16:37:54.6192386Z 2025-11-03T16:37:54.6192490Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6192837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6193147Z res = mod(**inputs) 2025-11-03T16:37:54.6193501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6193874Z outputs = self.model.decoder( 2025-11-03T16:37:54.6194336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6194719Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6195067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6195424Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6195782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6196132Z return func(*args, **kwargs) 2025-11-03T16:37:54.6196487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-11-03T16:37:54.6196857Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:54.6196983Z 2025-11-03T16:37:54.6197132Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6197464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6197764Z res = mod(**inputs) 2025-11-03T16:37:54.6198097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6198462Z outputs = self.model.decoder( 2025-11-03T16:37:54.6198824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6199215Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6199535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6199874Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6200231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6200586Z return func(*args, **kwargs) 2025-11-03T16:37:54.6200951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6201343Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6201726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6202083Z return func(*args, **kwargs) 2025-11-03T16:37:54.6202441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 402, in forward 2025-11-03T16:37:54.6202909Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:37:54.6203103Z 2025-11-03T16:37:54.6203202Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6203543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6203849Z res = mod(**inputs) 2025-11-03T16:37:54.6204198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6204573Z outputs = self.model.decoder( 2025-11-03T16:37:54.6204948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6205341Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6205682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6206040Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6206403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6206765Z return func(*args, **kwargs) 2025-11-03T16:37:54.6207136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6207540Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6207925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6208283Z return func(*args, **kwargs) 2025-11-03T16:37:54.6208681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 421, in forward 2025-11-03T16:37:54.6209067Z key_states = self.k_proj(current_states) 2025-11-03T16:37:54.6209199Z 2025-11-03T16:37:54.6209302Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6209631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6209933Z res = mod(**inputs) 2025-11-03T16:37:54.6210281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6210643Z outputs = self.model.decoder( 2025-11-03T16:37:54.6211001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6211359Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6211691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6212029Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6212383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6212799Z return func(*args, **kwargs) 2025-11-03T16:37:54.6213157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6213689Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6214064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6214423Z return func(*args, **kwargs) 2025-11-03T16:37:54.6214782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 422, in forward 2025-11-03T16:37:54.6215177Z value_states = self.v_proj(current_states) 2025-11-03T16:37:54.6215326Z 2025-11-03T16:37:54.6215407Z cudagraph partition due to non gpu ops 2025-11-03T16:37:54.6215640Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6216035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6216327Z res = mod(**inputs) 2025-11-03T16:37:54.6216672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6217039Z outputs = self.model.decoder( 2025-11-03T16:37:54.6217402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6217766Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6218095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6218438Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6218824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6219174Z return func(*args, **kwargs) 2025-11-03T16:37:54.6219521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6219912Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6220285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6220627Z return func(*args, **kwargs) 2025-11-03T16:37:54.6220978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 440, in forward 2025-11-03T16:37:54.6221357Z attn_output, attn_weights = attention_interface( 2025-11-03T16:37:54.6221801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:37:54.6222258Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:37:54.6222432Z 2025-11-03T16:37:54.6222536Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6222872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6223165Z res = mod(**inputs) 2025-11-03T16:37:54.6223506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6223873Z outputs = self.model.decoder( 2025-11-03T16:37:54.6224234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6224590Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6224916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6225253Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6225608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6225993Z return func(*args, **kwargs) 2025-11-03T16:37:54.6226343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 763, in forward 2025-11-03T16:37:54.6226733Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:37:54.6227104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6227448Z return func(*args, **kwargs) 2025-11-03T16:37:54.6227798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 454, in forward 2025-11-03T16:37:54.6228174Z attn_output = self.out_proj(attn_output) 2025-11-03T16:37:54.6228309Z 2025-11-03T16:37:54.6228408Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6228743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6229068Z res = mod(**inputs) 2025-11-03T16:37:54.6229406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6229779Z outputs = self.model.decoder( 2025-11-03T16:37:54.6230141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6230510Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6230840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6231177Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6231576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6231954Z return func(*args, **kwargs) 2025-11-03T16:37:54.6232330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6232754Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6232927Z 2025-11-03T16:37:54.6233026Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6233378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6233687Z res = mod(**inputs) 2025-11-03T16:37:54.6234119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6234511Z outputs = self.model.decoder( 2025-11-03T16:37:54.6234913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6235297Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6235638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6235996Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6236360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6236727Z return func(*args, **kwargs) 2025-11-03T16:37:54.6237094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 795, in forward 2025-11-03T16:37:54.6237514Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:37:54.6237882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:37:54.6238217Z return self.act(input) 2025-11-03T16:37:54.6238331Z 2025-11-03T16:37:54.6238434Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6238782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6239113Z res = mod(**inputs) 2025-11-03T16:37:54.6239466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1679, in forward 2025-11-03T16:37:54.6239857Z outputs = self.model.decoder( 2025-11-03T16:37:54.6240235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1038, in forward 2025-11-03T16:37:54.6240617Z layer_outputs = decoder_layer( 2025-11-03T16:37:54.6240957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:37:54.6241307Z return super().__call__(*args, **kwargs) 2025-11-03T16:37:54.6241680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:37:54.6242044Z return func(*args, **kwargs) 2025-11-03T16:37:54.6242418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 797, in forward 2025-11-03T16:37:54.6242818Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:37:54.6242958Z 2025-11-03T16:37:54.6243057Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6243401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6243707Z res = mod(**inputs) 2025-11-03T16:37:54.6244058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1695, in forward 2025-11-03T16:37:54.6244438Z logits = self.lm_head(outputs[0]) 2025-11-03T16:37:54.6244569Z 2025-11-03T16:37:54.6244665Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:37:54.6245005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:37:54.6245330Z res = mod(**inputs) 2025-11-03T16:37:54.6245683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1701, in forward 2025-11-03T16:37:54.6246124Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:37:54.6246319Z 2025-11-03T16:38:03.4873062Z Compilation time (from dynamo_timed): 12.079831489 2025-11-03T16:38:03.5252966Z pass 2025-11-03T16:38:03.5257043Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:03.5259144Z TIMING: _recursive_pre_grad_passes:0.00504 _recursive_joint_graph_passes:0.2431 _recursive_post_grad_passes:0.04093 async_compile.wait:0.686 code_gen:8.25483 inductor_compile:8.93349 backend_compile:10.6271 gc:0.00116 entire_frame_compile:12.07983 total_wall_time:12.07983 2025-11-03T16:38:03.5260468Z STATS: call_* op count: 180 | FakeTensorMode.__torch_dispatch__:4030 | FakeTensor.__torch_dispatch__:2342 | ProxyTorchDispatchMode.__torch_dispatch__:1079 2025-11-03T16:38:03.5261552Z Dynamo produced 1 graphs covering 180 ops with 0 graph breaks (0 unique) 2025-11-03T16:38:05.9960495Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:38:05.9961283Z import pynvml # type: ignore[import] 2025-11-03T16:38:09.1769541Z 2025-11-03T16:38:12.1197571Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:38:12.1201728Z loading model: 0it [00:02, ?it/s] 2025-11-03T16:38:12.1214981Z cpu eval PegasusForCausalLM 2025-11-03T16:38:12.4817210Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:12.6316683Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:12.7465865Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:19.4756848Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4761698Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4763815Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4764197Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4769468Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4771413Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4771855Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4776706Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4777145Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4777446Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4777753Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4777967Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4778837Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4779296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4779994Z res = mod(**inputs) 2025-11-03T16:38:19.4781952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4782656Z outputs = self.model.decoder( 2025-11-03T16:38:19.4783284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4787602Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4793311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4797915Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4800174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4800873Z return func(*args, **kwargs) 2025-11-03T16:38:19.4805515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4811186Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4816159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4820855Z return func(*args, **kwargs) 2025-11-03T16:38:19.4821374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.4821872Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.4822080Z 2025-11-03T16:38:19.4822192Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4822731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4823068Z res = mod(**inputs) 2025-11-03T16:38:19.4823455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4823873Z outputs = self.model.decoder( 2025-11-03T16:38:19.4824262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4824660Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4825012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4825375Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4825753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4826113Z return func(*args, **kwargs) 2025-11-03T16:38:19.4826495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4826924Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4827391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4827747Z return func(*args, **kwargs) 2025-11-03T16:38:19.4828127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.4828534Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.4828671Z 2025-11-03T16:38:19.4828787Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4829148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4829475Z res = mod(**inputs) 2025-11-03T16:38:19.4829855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4830244Z outputs = self.model.decoder( 2025-11-03T16:38:19.4830669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4831055Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4831389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4831739Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4832122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4832493Z return func(*args, **kwargs) 2025-11-03T16:38:19.4832961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4833373Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4833808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4834319Z return func(*args, **kwargs) 2025-11-03T16:38:19.4834734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.4835171Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.4835335Z 2025-11-03T16:38:19.4835415Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4835651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4836003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4836315Z res = mod(**inputs) 2025-11-03T16:38:19.4836672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4837112Z outputs = self.model.decoder( 2025-11-03T16:38:19.4837496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4837878Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4838206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4838556Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4838921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4839286Z return func(*args, **kwargs) 2025-11-03T16:38:19.4839661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4840069Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4840467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4840839Z return func(*args, **kwargs) 2025-11-03T16:38:19.4841210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.4841669Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.4842109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.4842599Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.4842787Z 2025-11-03T16:38:19.4842887Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4843234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4843545Z res = mod(**inputs) 2025-11-03T16:38:19.4843909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4844309Z outputs = self.model.decoder( 2025-11-03T16:38:19.4844714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4845106Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4845446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4845825Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4846213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4846577Z return func(*args, **kwargs) 2025-11-03T16:38:19.4846948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4847343Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4847746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4848107Z return func(*args, **kwargs) 2025-11-03T16:38:19.4848473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.4848860Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.4848989Z 2025-11-03T16:38:19.4849088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4849430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4849736Z res = mod(**inputs) 2025-11-03T16:38:19.4850096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4850468Z outputs = self.model.decoder( 2025-11-03T16:38:19.4850860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4851240Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4851577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4851921Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4852277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4852633Z return func(*args, **kwargs) 2025-11-03T16:38:19.4852996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4853416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4853584Z 2025-11-03T16:38:19.4853690Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4854023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4854329Z res = mod(**inputs) 2025-11-03T16:38:19.4854683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4855080Z outputs = self.model.decoder( 2025-11-03T16:38:19.4855449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4855826Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4856159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4856505Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4856866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4857215Z return func(*args, **kwargs) 2025-11-03T16:38:19.4857584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4858034Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4858409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.4858742Z return self.act(input) 2025-11-03T16:38:19.4858848Z 2025-11-03T16:38:19.4858948Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4859291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4859599Z res = mod(**inputs) 2025-11-03T16:38:19.4859985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4860371Z outputs = self.model.decoder( 2025-11-03T16:38:19.4860762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4861141Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4861478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4861825Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4862187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4862534Z return func(*args, **kwargs) 2025-11-03T16:38:19.4862902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.4863283Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.4863415Z 2025-11-03T16:38:19.4863519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4863877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4864181Z res = mod(**inputs) 2025-11-03T16:38:19.4864536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4864917Z outputs = self.model.decoder( 2025-11-03T16:38:19.4865288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4865665Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4865992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4866338Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4866707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4867055Z return func(*args, **kwargs) 2025-11-03T16:38:19.4867406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4867795Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4868202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4868553Z return func(*args, **kwargs) 2025-11-03T16:38:19.4868906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.4869337Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.4869536Z 2025-11-03T16:38:19.4869632Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4869961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4870258Z res = mod(**inputs) 2025-11-03T16:38:19.4870609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4870974Z outputs = self.model.decoder( 2025-11-03T16:38:19.4871372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4871758Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4872101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4872446Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4872817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4873179Z return func(*args, **kwargs) 2025-11-03T16:38:19.4873552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4874002Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4874520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4874931Z return func(*args, **kwargs) 2025-11-03T16:38:19.4875345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.4875773Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.4875910Z 2025-11-03T16:38:19.4876022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4876369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4876697Z res = mod(**inputs) 2025-11-03T16:38:19.4877075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4877496Z outputs = self.model.decoder( 2025-11-03T16:38:19.4877881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4878280Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4878625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4878982Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4879355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4879713Z return func(*args, **kwargs) 2025-11-03T16:38:19.4880085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4880498Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4880893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4881256Z return func(*args, **kwargs) 2025-11-03T16:38:19.4881622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.4882044Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.4882192Z 2025-11-03T16:38:19.4882276Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4882513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4882861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4883179Z res = mod(**inputs) 2025-11-03T16:38:19.4883547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4883937Z outputs = self.model.decoder( 2025-11-03T16:38:19.4884327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4884708Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4885073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4885434Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4885804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4886161Z return func(*args, **kwargs) 2025-11-03T16:38:19.4886534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4886944Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4887336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4887699Z return func(*args, **kwargs) 2025-11-03T16:38:19.4888086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.4888508Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.4888949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.4889424Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.4889604Z 2025-11-03T16:38:19.4889716Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4890061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4890377Z res = mod(**inputs) 2025-11-03T16:38:19.4890743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4891151Z outputs = self.model.decoder( 2025-11-03T16:38:19.4891544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4891934Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4892288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4892653Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4893038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4893406Z return func(*args, **kwargs) 2025-11-03T16:38:19.4893794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4894218Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4894613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4894967Z return func(*args, **kwargs) 2025-11-03T16:38:19.4895323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.4895719Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.4895853Z 2025-11-03T16:38:19.4895949Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4896279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4896575Z res = mod(**inputs) 2025-11-03T16:38:19.4896913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4897290Z outputs = self.model.decoder( 2025-11-03T16:38:19.4897657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4898026Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4898344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4899611Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4899967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4900314Z return func(*args, **kwargs) 2025-11-03T16:38:19.4900674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4901076Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4901243Z 2025-11-03T16:38:19.4901339Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4901671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4901974Z res = mod(**inputs) 2025-11-03T16:38:19.4902336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4902716Z outputs = self.model.decoder( 2025-11-03T16:38:19.4903091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4903470Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4903795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4904134Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4904494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4904844Z return func(*args, **kwargs) 2025-11-03T16:38:19.4905218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4905627Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4905988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.4906315Z return self.act(input) 2025-11-03T16:38:19.4906427Z 2025-11-03T16:38:19.4906526Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4906859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4907158Z res = mod(**inputs) 2025-11-03T16:38:19.4907499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4907876Z outputs = self.model.decoder( 2025-11-03T16:38:19.4908244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4908615Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4908935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4909296Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4909650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4910001Z return func(*args, **kwargs) 2025-11-03T16:38:19.4910359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.4910731Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.4910874Z 2025-11-03T16:38:19.4910973Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4911315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4911622Z res = mod(**inputs) 2025-11-03T16:38:19.4911973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4912376Z outputs = self.model.decoder( 2025-11-03T16:38:19.4912758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4913141Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4913727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4914133Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4914522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4914890Z return func(*args, **kwargs) 2025-11-03T16:38:19.4915273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4915729Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4916124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4916483Z return func(*args, **kwargs) 2025-11-03T16:38:19.4916848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.4917299Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.4917493Z 2025-11-03T16:38:19.4917601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4917939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4918250Z res = mod(**inputs) 2025-11-03T16:38:19.4918609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4919036Z outputs = self.model.decoder( 2025-11-03T16:38:19.4919411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4919791Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4920126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4920477Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4920841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4921192Z return func(*args, **kwargs) 2025-11-03T16:38:19.4921558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4921962Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4922347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4922703Z return func(*args, **kwargs) 2025-11-03T16:38:19.4923063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.4923479Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.4923609Z 2025-11-03T16:38:19.4923716Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4924057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4924360Z res = mod(**inputs) 2025-11-03T16:38:19.4924722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4925107Z outputs = self.model.decoder( 2025-11-03T16:38:19.4925489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4925872Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4926209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4926577Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4926939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4927299Z return func(*args, **kwargs) 2025-11-03T16:38:19.4927650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4928043Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4928425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4928781Z return func(*args, **kwargs) 2025-11-03T16:38:19.4929165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.4929554Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.4929699Z 2025-11-03T16:38:19.4929776Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4930006Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4930350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4930658Z res = mod(**inputs) 2025-11-03T16:38:19.4931009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4931394Z outputs = self.model.decoder( 2025-11-03T16:38:19.4931770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4932151Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4932496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4932846Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4933214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4933575Z return func(*args, **kwargs) 2025-11-03T16:38:19.4933939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4934335Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4934719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4935078Z return func(*args, **kwargs) 2025-11-03T16:38:19.4935450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.4935841Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.4936250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.4936722Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.4936901Z 2025-11-03T16:38:19.4936997Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4937338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4937645Z res = mod(**inputs) 2025-11-03T16:38:19.4937999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4938374Z outputs = self.model.decoder( 2025-11-03T16:38:19.4938744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4939117Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4939436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4939793Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4940149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4940496Z return func(*args, **kwargs) 2025-11-03T16:38:19.4940852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4941237Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4941612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4941957Z return func(*args, **kwargs) 2025-11-03T16:38:19.4942331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.4942711Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.4942847Z 2025-11-03T16:38:19.4942945Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4943280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4943583Z res = mod(**inputs) 2025-11-03T16:38:19.4943933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4944299Z outputs = self.model.decoder( 2025-11-03T16:38:19.4944665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4945039Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4945383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4945727Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4946085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4946441Z return func(*args, **kwargs) 2025-11-03T16:38:19.4946807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4947224Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4947391Z 2025-11-03T16:38:19.4947497Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4947829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4948137Z res = mod(**inputs) 2025-11-03T16:38:19.4948491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4948870Z outputs = self.model.decoder( 2025-11-03T16:38:19.4949235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4949622Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4949953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4950290Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4950645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4950985Z return func(*args, **kwargs) 2025-11-03T16:38:19.4951342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4951748Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4952113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.4952424Z return self.act(input) 2025-11-03T16:38:19.4952552Z 2025-11-03T16:38:19.4952646Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4952984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4953284Z res = mod(**inputs) 2025-11-03T16:38:19.4953632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4954014Z outputs = self.model.decoder( 2025-11-03T16:38:19.4954477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4954874Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4955225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4955599Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4955964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4956342Z return func(*args, **kwargs) 2025-11-03T16:38:19.4956724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.4957125Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.4957261Z 2025-11-03T16:38:19.4957365Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4957719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4958034Z res = mod(**inputs) 2025-11-03T16:38:19.4958403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4958814Z outputs = self.model.decoder( 2025-11-03T16:38:19.4959198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4959597Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4959946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4960304Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4960678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4961040Z return func(*args, **kwargs) 2025-11-03T16:38:19.4961415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-11-03T16:38:19.4961810Z hidden_states = residual + hidden_states 2025-11-03T16:38:19.4961945Z 2025-11-03T16:38:19.4962056Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4962401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4962739Z res = mod(**inputs) 2025-11-03T16:38:19.4963107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4963505Z outputs = self.model.decoder( 2025-11-03T16:38:19.4963890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4964276Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4964625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4964968Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4965324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4965679Z return func(*args, **kwargs) 2025-11-03T16:38:19.4966036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4966465Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4966870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4967234Z return func(*args, **kwargs) 2025-11-03T16:38:19.4967602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.4968063Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.4968269Z 2025-11-03T16:38:19.4968373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4968723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4969074Z res = mod(**inputs) 2025-11-03T16:38:19.4969429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4969818Z outputs = self.model.decoder( 2025-11-03T16:38:19.4970194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4970578Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4970911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4971253Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4971617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4971973Z return func(*args, **kwargs) 2025-11-03T16:38:19.4972360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4972762Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4973151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4973509Z return func(*args, **kwargs) 2025-11-03T16:38:19.4973876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.4974262Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.4974393Z 2025-11-03T16:38:19.4974493Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4974838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4975149Z res = mod(**inputs) 2025-11-03T16:38:19.4975507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4975887Z outputs = self.model.decoder( 2025-11-03T16:38:19.4976259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4976662Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4976997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4977344Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4977700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4978057Z return func(*args, **kwargs) 2025-11-03T16:38:19.4978424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4978828Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4979210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4979578Z return func(*args, **kwargs) 2025-11-03T16:38:19.4979943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.4980337Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.4980474Z 2025-11-03T16:38:19.4980562Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.4980792Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4981127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4981437Z res = mod(**inputs) 2025-11-03T16:38:19.4981797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4982184Z outputs = self.model.decoder( 2025-11-03T16:38:19.4982568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4982952Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4983322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4983678Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4984059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4984409Z return func(*args, **kwargs) 2025-11-03T16:38:19.4984771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4985169Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4985562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4985919Z return func(*args, **kwargs) 2025-11-03T16:38:19.4986288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.4986694Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.4987125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.4987592Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.4987767Z 2025-11-03T16:38:19.4987866Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4988208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4988518Z res = mod(**inputs) 2025-11-03T16:38:19.4988882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4989266Z outputs = self.model.decoder( 2025-11-03T16:38:19.4989641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4990039Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4990370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4990717Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4991078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4991428Z return func(*args, **kwargs) 2025-11-03T16:38:19.4991797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.4992202Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.4992587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4992960Z return func(*args, **kwargs) 2025-11-03T16:38:19.4993332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.4993729Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.4993856Z 2025-11-03T16:38:19.4993961Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4994370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4994685Z res = mod(**inputs) 2025-11-03T16:38:19.4995075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.4995468Z outputs = self.model.decoder( 2025-11-03T16:38:19.4995873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.4996255Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.4996585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.4996942Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.4997301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.4997648Z return func(*args, **kwargs) 2025-11-03T16:38:19.4998000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.4998412Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.4998582Z 2025-11-03T16:38:19.4998680Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.4999243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.4999552Z res = mod(**inputs) 2025-11-03T16:38:19.4999903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5000288Z outputs = self.model.decoder( 2025-11-03T16:38:19.5000671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5001054Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5001388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5001727Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5002089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5002446Z return func(*args, **kwargs) 2025-11-03T16:38:19.5002815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5003233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5003625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5003956Z return self.act(input) 2025-11-03T16:38:19.5004061Z 2025-11-03T16:38:19.5004172Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5004524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5004834Z res = mod(**inputs) 2025-11-03T16:38:19.5005199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5005595Z outputs = self.model.decoder( 2025-11-03T16:38:19.5005981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5006376Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5006721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5007075Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5007444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5007801Z return func(*args, **kwargs) 2025-11-03T16:38:19.5008163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5008559Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5008700Z 2025-11-03T16:38:19.5008799Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5009150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5009464Z res = mod(**inputs) 2025-11-03T16:38:19.5009806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5010185Z outputs = self.model.decoder( 2025-11-03T16:38:19.5010548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5010917Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5011240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5011573Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5011929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5012275Z return func(*args, **kwargs) 2025-11-03T16:38:19.5012649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5013051Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5013662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5014018Z return func(*args, **kwargs) 2025-11-03T16:38:19.5014388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5014851Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5015049Z 2025-11-03T16:38:19.5015150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5015508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5015815Z res = mod(**inputs) 2025-11-03T16:38:19.5016168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5016544Z outputs = self.model.decoder( 2025-11-03T16:38:19.5016942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5017311Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5017639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5017979Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5018324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5018675Z return func(*args, **kwargs) 2025-11-03T16:38:19.5019035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5019434Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5019809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5020191Z return func(*args, **kwargs) 2025-11-03T16:38:19.5020554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5020930Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5021058Z 2025-11-03T16:38:19.5021164Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5021498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5021796Z res = mod(**inputs) 2025-11-03T16:38:19.5022144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5022516Z outputs = self.model.decoder( 2025-11-03T16:38:19.5022912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5023285Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5023613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5023956Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5024323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5024673Z return func(*args, **kwargs) 2025-11-03T16:38:19.5025024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5025420Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5025817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5026167Z return func(*args, **kwargs) 2025-11-03T16:38:19.5026528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5026909Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5027054Z 2025-11-03T16:38:19.5027131Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5027356Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5027693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5027987Z res = mod(**inputs) 2025-11-03T16:38:19.5028336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5028708Z outputs = self.model.decoder( 2025-11-03T16:38:19.5029076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5029447Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5029768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5030167Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5030523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5030871Z return func(*args, **kwargs) 2025-11-03T16:38:19.5031221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5031613Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5031986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5032333Z return func(*args, **kwargs) 2025-11-03T16:38:19.5032691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5033098Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5033523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5033992Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5034236Z 2025-11-03T16:38:19.5034350Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5034706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5035017Z res = mod(**inputs) 2025-11-03T16:38:19.5035418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5035795Z outputs = self.model.decoder( 2025-11-03T16:38:19.5036205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5036608Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5036951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5037317Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5037700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5038073Z return func(*args, **kwargs) 2025-11-03T16:38:19.5038450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5038871Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5039288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5039657Z return func(*args, **kwargs) 2025-11-03T16:38:19.5040034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5040430Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5040571Z 2025-11-03T16:38:19.5040672Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5041023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5041347Z res = mod(**inputs) 2025-11-03T16:38:19.5041717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5042104Z outputs = self.model.decoder( 2025-11-03T16:38:19.5042490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5042876Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5043218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5043590Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5043966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5044341Z return func(*args, **kwargs) 2025-11-03T16:38:19.5044698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5045105Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5045267Z 2025-11-03T16:38:19.5045364Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5045698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5046000Z res = mod(**inputs) 2025-11-03T16:38:19.5046350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5046743Z outputs = self.model.decoder( 2025-11-03T16:38:19.5047103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5047471Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5047795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5048140Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5048491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5048842Z return func(*args, **kwargs) 2025-11-03T16:38:19.5049201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5049625Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5049988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5050304Z return self.act(input) 2025-11-03T16:38:19.5050414Z 2025-11-03T16:38:19.5050510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5050843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5051142Z res = mod(**inputs) 2025-11-03T16:38:19.5051479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5051848Z outputs = self.model.decoder( 2025-11-03T16:38:19.5052211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5052592Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5052920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5053255Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5053612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5053959Z return func(*args, **kwargs) 2025-11-03T16:38:19.5054314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5054688Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5054817Z 2025-11-03T16:38:19.5054912Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5055243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5055544Z res = mod(**inputs) 2025-11-03T16:38:19.5055893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5056257Z outputs = self.model.decoder( 2025-11-03T16:38:19.5056646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5057015Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5057348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5057688Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5058046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5058393Z return func(*args, **kwargs) 2025-11-03T16:38:19.5058754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-11-03T16:38:19.5059128Z hidden_states = residual + hidden_states 2025-11-03T16:38:19.5059256Z 2025-11-03T16:38:19.5059361Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5059705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5060006Z res = mod(**inputs) 2025-11-03T16:38:19.5060353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5060726Z outputs = self.model.decoder( 2025-11-03T16:38:19.5061084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5061456Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5061781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5062122Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5062493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5062836Z return func(*args, **kwargs) 2025-11-03T16:38:19.5063193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5063586Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5063955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5064300Z return func(*args, **kwargs) 2025-11-03T16:38:19.5064647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5065085Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5065280Z 2025-11-03T16:38:19.5065392Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5065729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5066024Z res = mod(**inputs) 2025-11-03T16:38:19.5066371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5066745Z outputs = self.model.decoder( 2025-11-03T16:38:19.5067111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5067479Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5067794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5068134Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5068485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5068834Z return func(*args, **kwargs) 2025-11-03T16:38:19.5069187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5069611Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5069984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5070332Z return func(*args, **kwargs) 2025-11-03T16:38:19.5070689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5071060Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5071193Z 2025-11-03T16:38:19.5071289Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5071625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5071929Z res = mod(**inputs) 2025-11-03T16:38:19.5072285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5072688Z outputs = self.model.decoder( 2025-11-03T16:38:19.5073072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5073459Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5073797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5074217Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5074580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5074941Z return func(*args, **kwargs) 2025-11-03T16:38:19.5075322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5075764Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5076151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5076525Z return func(*args, **kwargs) 2025-11-03T16:38:19.5076902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5077319Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5077460Z 2025-11-03T16:38:19.5077549Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5077780Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5078132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5078454Z res = mod(**inputs) 2025-11-03T16:38:19.5078843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5079239Z outputs = self.model.decoder( 2025-11-03T16:38:19.5079617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5080009Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5080353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5080711Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5081079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5081446Z return func(*args, **kwargs) 2025-11-03T16:38:19.5081821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5082236Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5082634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5083019Z return func(*args, **kwargs) 2025-11-03T16:38:19.5083396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5083817Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5084247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5084711Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5084886Z 2025-11-03T16:38:19.5084984Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5085327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5085636Z res = mod(**inputs) 2025-11-03T16:38:19.5085999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5086403Z outputs = self.model.decoder( 2025-11-03T16:38:19.5086781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5087166Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5087503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5087857Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5088217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5088579Z return func(*args, **kwargs) 2025-11-03T16:38:19.5088947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5089368Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5089758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5090103Z return func(*args, **kwargs) 2025-11-03T16:38:19.5090461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5090838Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5090965Z 2025-11-03T16:38:19.5091068Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5091391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5091687Z res = mod(**inputs) 2025-11-03T16:38:19.5092033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5092419Z outputs = self.model.decoder( 2025-11-03T16:38:19.5092784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5093160Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5093491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5093833Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5094197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5094553Z return func(*args, **kwargs) 2025-11-03T16:38:19.5094909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5095327Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5095496Z 2025-11-03T16:38:19.5095597Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5095935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5096261Z res = mod(**inputs) 2025-11-03T16:38:19.5096610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5096986Z outputs = self.model.decoder( 2025-11-03T16:38:19.5097353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5097721Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5098037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5098381Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5098740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5099092Z return func(*args, **kwargs) 2025-11-03T16:38:19.5099453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5099876Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5100247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5100572Z return self.act(input) 2025-11-03T16:38:19.5100675Z 2025-11-03T16:38:19.5100778Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5101102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5101405Z res = mod(**inputs) 2025-11-03T16:38:19.5101751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5102124Z outputs = self.model.decoder( 2025-11-03T16:38:19.5102504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5102873Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5103199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5103541Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5103901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5104249Z return func(*args, **kwargs) 2025-11-03T16:38:19.5104603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5104982Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5105111Z 2025-11-03T16:38:19.5105234Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5105572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5105867Z res = mod(**inputs) 2025-11-03T16:38:19.5106215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5106585Z outputs = self.model.decoder( 2025-11-03T16:38:19.5106949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5107315Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5107630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5107965Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5108319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5108668Z return func(*args, **kwargs) 2025-11-03T16:38:19.5109019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5109432Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5109806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5110154Z return func(*args, **kwargs) 2025-11-03T16:38:19.5110510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5110943Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5111138Z 2025-11-03T16:38:19.5111235Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5111581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5111890Z res = mod(**inputs) 2025-11-03T16:38:19.5112249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5112650Z outputs = self.model.decoder( 2025-11-03T16:38:19.5113030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5113583Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5113924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5114324Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5114698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5115080Z return func(*args, **kwargs) 2025-11-03T16:38:19.5115495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5115909Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5116289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5116645Z return func(*args, **kwargs) 2025-11-03T16:38:19.5117010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5117400Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5117530Z 2025-11-03T16:38:19.5117637Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5117975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5118285Z res = mod(**inputs) 2025-11-03T16:38:19.5118679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5119064Z outputs = self.model.decoder( 2025-11-03T16:38:19.5119443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5119824Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5120158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5120505Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5120870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5121222Z return func(*args, **kwargs) 2025-11-03T16:38:19.5121588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5121995Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5122384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5122744Z return func(*args, **kwargs) 2025-11-03T16:38:19.5123125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5123524Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5123669Z 2025-11-03T16:38:19.5123748Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5123980Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5124311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5124666Z res = mod(**inputs) 2025-11-03T16:38:19.5125022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5125401Z outputs = self.model.decoder( 2025-11-03T16:38:19.5125777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5126178Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5126515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5126865Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5127238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5127608Z return func(*args, **kwargs) 2025-11-03T16:38:19.5127963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5128366Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5128743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5129116Z return func(*args, **kwargs) 2025-11-03T16:38:19.5129470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5129866Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5130285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5130742Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5130913Z 2025-11-03T16:38:19.5131017Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5131344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5131645Z res = mod(**inputs) 2025-11-03T16:38:19.5132009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5132391Z outputs = self.model.decoder( 2025-11-03T16:38:19.5132764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5133134Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5133464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5133807Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5134167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5134520Z return func(*args, **kwargs) 2025-11-03T16:38:19.5134877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5135272Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5135647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5136002Z return func(*args, **kwargs) 2025-11-03T16:38:19.5136370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5136747Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5136882Z 2025-11-03T16:38:19.5136981Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5137312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5137612Z res = mod(**inputs) 2025-11-03T16:38:19.5137953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5138331Z outputs = self.model.decoder( 2025-11-03T16:38:19.5138700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5139070Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5139407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5139750Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5140103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5140449Z return func(*args, **kwargs) 2025-11-03T16:38:19.5140802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5141203Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5141370Z 2025-11-03T16:38:19.5141468Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5141801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5142114Z res = mod(**inputs) 2025-11-03T16:38:19.5142464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5142832Z outputs = self.model.decoder( 2025-11-03T16:38:19.5143195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5143566Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5143893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5144222Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5144581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5144930Z return func(*args, **kwargs) 2025-11-03T16:38:19.5145305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5145723Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5146094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5146426Z return self.act(input) 2025-11-03T16:38:19.5146538Z 2025-11-03T16:38:19.5146637Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5146982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5147289Z res = mod(**inputs) 2025-11-03T16:38:19.5147637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5148016Z outputs = self.model.decoder( 2025-11-03T16:38:19.5148393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5148774Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5149101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5149459Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5149815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5150165Z return func(*args, **kwargs) 2025-11-03T16:38:19.5150523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5150894Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5151029Z 2025-11-03T16:38:19.5151124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5151459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5151761Z res = mod(**inputs) 2025-11-03T16:38:19.5152114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5152517Z outputs = self.model.decoder( 2025-11-03T16:38:19.5152893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5153284Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5153623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5153976Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5154464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5154873Z return func(*args, **kwargs) 2025-11-03T16:38:19.5155285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-11-03T16:38:19.5155687Z hidden_states = residual + hidden_states 2025-11-03T16:38:19.5155823Z 2025-11-03T16:38:19.5155926Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5156317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5156630Z res = mod(**inputs) 2025-11-03T16:38:19.5156994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5157385Z outputs = self.model.decoder( 2025-11-03T16:38:19.5157758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5158144Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5158497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5158851Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5159209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5159575Z return func(*args, **kwargs) 2025-11-03T16:38:19.5159944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5160355Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5160739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5161092Z return func(*args, **kwargs) 2025-11-03T16:38:19.5161464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5161920Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5162115Z 2025-11-03T16:38:19.5162224Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5162569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5162894Z res = mod(**inputs) 2025-11-03T16:38:19.5163255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5163638Z outputs = self.model.decoder( 2025-11-03T16:38:19.5164018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5164396Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5164731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5165077Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5165444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5165802Z return func(*args, **kwargs) 2025-11-03T16:38:19.5166177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5166576Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5166956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5167309Z return func(*args, **kwargs) 2025-11-03T16:38:19.5167671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5168046Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5168183Z 2025-11-03T16:38:19.5168281Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5168647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5168957Z res = mod(**inputs) 2025-11-03T16:38:19.5169310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5169695Z outputs = self.model.decoder( 2025-11-03T16:38:19.5170070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5170450Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5170788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5171129Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5171492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5171851Z return func(*args, **kwargs) 2025-11-03T16:38:19.5172239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5172656Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5173046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5173409Z return func(*args, **kwargs) 2025-11-03T16:38:19.5173786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5174195Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5174335Z 2025-11-03T16:38:19.5174417Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5174653Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5175003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5175324Z res = mod(**inputs) 2025-11-03T16:38:19.5175698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5176102Z outputs = self.model.decoder( 2025-11-03T16:38:19.5176486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5176876Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5177217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5177565Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5177941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5178320Z return func(*args, **kwargs) 2025-11-03T16:38:19.5178697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5179118Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5179514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5179880Z return func(*args, **kwargs) 2025-11-03T16:38:19.5180249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5180653Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5181083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5181544Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5181547Z 2025-11-03T16:38:19.5181653Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5181867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5181938Z res = mod(**inputs) 2025-11-03T16:38:19.5182194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5182269Z outputs = self.model.decoder( 2025-11-03T16:38:19.5182545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5182618Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5182862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5182943Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5183188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5183258Z return func(*args, **kwargs) 2025-11-03T16:38:19.5183537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5183643Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5183888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5183963Z return func(*args, **kwargs) 2025-11-03T16:38:19.5184208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5184286Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5184289Z 2025-11-03T16:38:19.5184396Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5184585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5184653Z res = mod(**inputs) 2025-11-03T16:38:19.5184905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5184976Z outputs = self.model.decoder( 2025-11-03T16:38:19.5185255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5185326Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5185545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5185621Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5185857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5185924Z return func(*args, **kwargs) 2025-11-03T16:38:19.5186171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5186296Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5186300Z 2025-11-03T16:38:19.5186399Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5186615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5186676Z res = mod(**inputs) 2025-11-03T16:38:19.5186927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5187005Z outputs = self.model.decoder( 2025-11-03T16:38:19.5187258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5187336Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5187548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5187631Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5187879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5187952Z return func(*args, **kwargs) 2025-11-03T16:38:19.5188209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5188323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5188544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5188610Z return self.act(input) 2025-11-03T16:38:19.5188614Z 2025-11-03T16:38:19.5188711Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5188908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5188969Z res = mod(**inputs) 2025-11-03T16:38:19.5189244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5189317Z outputs = self.model.decoder( 2025-11-03T16:38:19.5189573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5189643Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5189849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5189932Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5190162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5190234Z return func(*args, **kwargs) 2025-11-03T16:38:19.5190481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5190561Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5190564Z 2025-11-03T16:38:19.5190673Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5190862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5190945Z res = mod(**inputs) 2025-11-03T16:38:19.5191198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5191268Z outputs = self.model.decoder( 2025-11-03T16:38:19.5191534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5191601Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5191821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5191898Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5192141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5192207Z return func(*args, **kwargs) 2025-11-03T16:38:19.5192477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5192581Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5192809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5192880Z return func(*args, **kwargs) 2025-11-03T16:38:19.5193130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5193273Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5193276Z 2025-11-03T16:38:19.5193390Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5193589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5193658Z res = mod(**inputs) 2025-11-03T16:38:19.5193913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5193990Z outputs = self.model.decoder( 2025-11-03T16:38:19.5194307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5194378Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5194601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5194679Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5194918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5195007Z return func(*args, **kwargs) 2025-11-03T16:38:19.5195272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5195380Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5195610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5195684Z return func(*args, **kwargs) 2025-11-03T16:38:19.5195930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5196006Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5196018Z 2025-11-03T16:38:19.5196114Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5196295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5196363Z res = mod(**inputs) 2025-11-03T16:38:19.5196618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5196715Z outputs = self.model.decoder( 2025-11-03T16:38:19.5196963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5197031Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5197245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5197318Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5197552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5197618Z return func(*args, **kwargs) 2025-11-03T16:38:19.5197864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5197966Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5198187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5198291Z return func(*args, **kwargs) 2025-11-03T16:38:19.5198537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5198628Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5198631Z 2025-11-03T16:38:19.5198708Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5198804Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5198996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5199056Z res = mod(**inputs) 2025-11-03T16:38:19.5199316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5199403Z outputs = self.model.decoder( 2025-11-03T16:38:19.5199655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5199734Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5199939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5200022Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5200250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5200315Z return func(*args, **kwargs) 2025-11-03T16:38:19.5200565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5200660Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5200912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5200978Z return func(*args, **kwargs) 2025-11-03T16:38:19.5201230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5201322Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5201594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5201727Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5201730Z 2025-11-03T16:38:19.5201825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5202016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5202076Z res = mod(**inputs) 2025-11-03T16:38:19.5202326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5202404Z outputs = self.model.decoder( 2025-11-03T16:38:19.5202670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5202743Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5202948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5203029Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5203254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5203318Z return func(*args, **kwargs) 2025-11-03T16:38:19.5203570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5203661Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5203895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5203975Z return func(*args, **kwargs) 2025-11-03T16:38:19.5204220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5204304Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5204307Z 2025-11-03T16:38:19.5204403Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5204589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5204649Z res = mod(**inputs) 2025-11-03T16:38:19.5204899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5204976Z outputs = self.model.decoder( 2025-11-03T16:38:19.5205239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5205316Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5205523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5205604Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5205828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5205892Z return func(*args, **kwargs) 2025-11-03T16:38:19.5206149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5206261Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5206265Z 2025-11-03T16:38:19.5206414Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5206600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5206662Z res = mod(**inputs) 2025-11-03T16:38:19.5206917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5206986Z outputs = self.model.decoder( 2025-11-03T16:38:19.5207239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5207305Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5207516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5207590Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5207820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5207895Z return func(*args, **kwargs) 2025-11-03T16:38:19.5208142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5208278Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5208485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5208549Z return self.act(input) 2025-11-03T16:38:19.5208553Z 2025-11-03T16:38:19.5208654Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5208839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5208905Z res = mod(**inputs) 2025-11-03T16:38:19.5209157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5209227Z outputs = self.model.decoder( 2025-11-03T16:38:19.5209484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5209568Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5209779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5209852Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5210082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5210146Z return func(*args, **kwargs) 2025-11-03T16:38:19.5210386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5210471Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5210474Z 2025-11-03T16:38:19.5210569Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5210775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5210835Z res = mod(**inputs) 2025-11-03T16:38:19.5211083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5211157Z outputs = self.model.decoder( 2025-11-03T16:38:19.5211407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5211478Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5211686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5211764Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5211990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5212069Z return func(*args, **kwargs) 2025-11-03T16:38:19.5212320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-11-03T16:38:19.5212398Z hidden_states = residual + hidden_states 2025-11-03T16:38:19.5212401Z 2025-11-03T16:38:19.5212502Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5212679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5212739Z res = mod(**inputs) 2025-11-03T16:38:19.5212988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5213056Z outputs = self.model.decoder( 2025-11-03T16:38:19.5213463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5213535Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5213749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5213832Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5214094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5214170Z return func(*args, **kwargs) 2025-11-03T16:38:19.5214423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5214523Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5214752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5214816Z return func(*args, **kwargs) 2025-11-03T16:38:19.5215073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5215217Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5215241Z 2025-11-03T16:38:19.5215346Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5215530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5215590Z res = mod(**inputs) 2025-11-03T16:38:19.5215845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5215914Z outputs = self.model.decoder( 2025-11-03T16:38:19.5216169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5216238Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5216454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5216551Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5216778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5216856Z return func(*args, **kwargs) 2025-11-03T16:38:19.5217103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5217200Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5217425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5217488Z return func(*args, **kwargs) 2025-11-03T16:38:19.5217736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5217811Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5217814Z 2025-11-03T16:38:19.5217938Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5218122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5218191Z res = mod(**inputs) 2025-11-03T16:38:19.5218435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5218502Z outputs = self.model.decoder( 2025-11-03T16:38:19.5218755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5218822Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5219032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5219104Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5219328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5219401Z return func(*args, **kwargs) 2025-11-03T16:38:19.5219642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5219770Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5219991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5220054Z return func(*args, **kwargs) 2025-11-03T16:38:19.5220302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5220382Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5220386Z 2025-11-03T16:38:19.5220468Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5220562Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5220755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5220815Z res = mod(**inputs) 2025-11-03T16:38:19.5221079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5221154Z outputs = self.model.decoder( 2025-11-03T16:38:19.5221399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5221471Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5221674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5221746Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5221976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5222038Z return func(*args, **kwargs) 2025-11-03T16:38:19.5222304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5222400Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5222623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5222691Z return func(*args, **kwargs) 2025-11-03T16:38:19.5222932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5223033Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5223302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5223433Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5223436Z 2025-11-03T16:38:19.5223546Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5223732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5223801Z res = mod(**inputs) 2025-11-03T16:38:19.5224049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5224123Z outputs = self.model.decoder( 2025-11-03T16:38:19.5224369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5224434Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5224645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5224719Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5224954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5225020Z return func(*args, **kwargs) 2025-11-03T16:38:19.5225271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5225381Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5225603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5225676Z return func(*args, **kwargs) 2025-11-03T16:38:19.5225921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5226005Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5226008Z 2025-11-03T16:38:19.5226102Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5226287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5226357Z res = mod(**inputs) 2025-11-03T16:38:19.5226609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5226702Z outputs = self.model.decoder( 2025-11-03T16:38:19.5226954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5227030Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5227240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5227315Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5227561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5227625Z return func(*args, **kwargs) 2025-11-03T16:38:19.5227898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5228012Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5228018Z 2025-11-03T16:38:19.5228112Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5228301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5228361Z res = mod(**inputs) 2025-11-03T16:38:19.5228612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5228680Z outputs = self.model.decoder( 2025-11-03T16:38:19.5228926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5229001Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5229220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5229305Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5229530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5229604Z return func(*args, **kwargs) 2025-11-03T16:38:19.5229849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5229961Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5230170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5230236Z return self.act(input) 2025-11-03T16:38:19.5230239Z 2025-11-03T16:38:19.5230342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5230524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5230587Z res = mod(**inputs) 2025-11-03T16:38:19.5230843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5230930Z outputs = self.model.decoder( 2025-11-03T16:38:19.5231182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5231250Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5231462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5231536Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5231762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5231836Z return func(*args, **kwargs) 2025-11-03T16:38:19.5232079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5232165Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5232182Z 2025-11-03T16:38:19.5232280Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5232461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5232530Z res = mod(**inputs) 2025-11-03T16:38:19.5232779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5232854Z outputs = self.model.decoder( 2025-11-03T16:38:19.5233102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5233169Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5233385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5233476Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5233718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5233788Z return func(*args, **kwargs) 2025-11-03T16:38:19.5234046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5234199Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5234444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5234520Z return func(*args, **kwargs) 2025-11-03T16:38:19.5234770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5234924Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5234950Z 2025-11-03T16:38:19.5235054Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5235247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5235322Z res = mod(**inputs) 2025-11-03T16:38:19.5235585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5235666Z outputs = self.model.decoder( 2025-11-03T16:38:19.5235996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5236075Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5236293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5236372Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5236621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5236688Z return func(*args, **kwargs) 2025-11-03T16:38:19.5236949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5237059Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5237290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5237366Z return func(*args, **kwargs) 2025-11-03T16:38:19.5237613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5237696Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5237699Z 2025-11-03T16:38:19.5237795Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5237990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5238050Z res = mod(**inputs) 2025-11-03T16:38:19.5238298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5238392Z outputs = self.model.decoder( 2025-11-03T16:38:19.5238644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5238718Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5238928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5239002Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5239240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5239305Z return func(*args, **kwargs) 2025-11-03T16:38:19.5239576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5239672Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5239902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5239973Z return func(*args, **kwargs) 2025-11-03T16:38:19.5240220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5240310Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5240313Z 2025-11-03T16:38:19.5240390Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5240494Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5240679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5240764Z res = mod(**inputs) 2025-11-03T16:38:19.5241030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5241102Z outputs = self.model.decoder( 2025-11-03T16:38:19.5241366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5241436Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5241648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5241731Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5241964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5242038Z return func(*args, **kwargs) 2025-11-03T16:38:19.5242288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5242383Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5242620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5242701Z return func(*args, **kwargs) 2025-11-03T16:38:19.5242959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5243053Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5243342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5243475Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5243478Z 2025-11-03T16:38:19.5243577Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5243781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5243844Z res = mod(**inputs) 2025-11-03T16:38:19.5244108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5244197Z outputs = self.model.decoder( 2025-11-03T16:38:19.5244449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5244525Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5244735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5244817Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5245048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5245123Z return func(*args, **kwargs) 2025-11-03T16:38:19.5245388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5245481Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5245719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5245784Z return func(*args, **kwargs) 2025-11-03T16:38:19.5246049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5246125Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5246128Z 2025-11-03T16:38:19.5246224Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5246414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5246474Z res = mod(**inputs) 2025-11-03T16:38:19.5246742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5246810Z outputs = self.model.decoder( 2025-11-03T16:38:19.5247065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5247132Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5247342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5247420Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5247643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5247714Z return func(*args, **kwargs) 2025-11-03T16:38:19.5247958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5248071Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5248076Z 2025-11-03T16:38:19.5248178Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5248362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5248449Z res = mod(**inputs) 2025-11-03T16:38:19.5248693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5248762Z outputs = self.model.decoder( 2025-11-03T16:38:19.5249013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5249079Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5249289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5249362Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5249597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5249659Z return func(*args, **kwargs) 2025-11-03T16:38:19.5249920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5250037Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5250237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5250308Z return self.act(input) 2025-11-03T16:38:19.5250311Z 2025-11-03T16:38:19.5250404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5250586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5250652Z res = mod(**inputs) 2025-11-03T16:38:19.5250914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5250990Z outputs = self.model.decoder( 2025-11-03T16:38:19.5251236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5251310Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5251514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5251585Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5251816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5251879Z return func(*args, **kwargs) 2025-11-03T16:38:19.5252129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5252220Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5252223Z 2025-11-03T16:38:19.5252320Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5252508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5252569Z res = mod(**inputs) 2025-11-03T16:38:19.5252823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5252892Z outputs = self.model.decoder( 2025-11-03T16:38:19.5253138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5253210Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5253414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5253494Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5253721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5253793Z return func(*args, **kwargs) 2025-11-03T16:38:19.5254037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-11-03T16:38:19.5254131Z hidden_states = residual + hidden_states 2025-11-03T16:38:19.5254135Z 2025-11-03T16:38:19.5254235Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5254418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5254484Z res = mod(**inputs) 2025-11-03T16:38:19.5254729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5254795Z outputs = self.model.decoder( 2025-11-03T16:38:19.5255050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5255117Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5255328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5255415Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5255645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5255709Z return func(*args, **kwargs) 2025-11-03T16:38:19.5255948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5256048Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5256271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5256344Z return func(*args, **kwargs) 2025-11-03T16:38:19.5256600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 217, in forward 2025-11-03T16:38:19.5256741Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-11-03T16:38:19.5256745Z 2025-11-03T16:38:19.5256847Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5257028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5257095Z res = mod(**inputs) 2025-11-03T16:38:19.5257339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5257407Z outputs = self.model.decoder( 2025-11-03T16:38:19.5257660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5257725Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5257954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5258030Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5258267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5258330Z return func(*args, **kwargs) 2025-11-03T16:38:19.5258573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5258670Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5258894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5258964Z return func(*args, **kwargs) 2025-11-03T16:38:19.5259205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-11-03T16:38:19.5259283Z key_states = self.k_proj(current_states) 2025-11-03T16:38:19.5259293Z 2025-11-03T16:38:19.5259386Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5259586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5259653Z res = mod(**inputs) 2025-11-03T16:38:19.5259896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5259970Z outputs = self.model.decoder( 2025-11-03T16:38:19.5260213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5260282Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5260493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5260567Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5260797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5260878Z return func(*args, **kwargs) 2025-11-03T16:38:19.5261119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5261220Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5261443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5261515Z return func(*args, **kwargs) 2025-11-03T16:38:19.5261756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-11-03T16:38:19.5261838Z value_states = self.v_proj(current_states) 2025-11-03T16:38:19.5261848Z 2025-11-03T16:38:19.5261924Z cudagraph partition due to non gpu ops 2025-11-03T16:38:19.5262044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5262240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5262303Z res = mod(**inputs) 2025-11-03T16:38:19.5262555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5262624Z outputs = self.model.decoder( 2025-11-03T16:38:19.5262871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5262945Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5263151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5263230Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5263472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5263539Z return func(*args, **kwargs) 2025-11-03T16:38:19.5263791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5263883Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5264111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5264174Z return func(*args, **kwargs) 2025-11-03T16:38:19.5264421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-11-03T16:38:19.5264513Z attn_output, attn_weights = attention_interface( 2025-11-03T16:38:19.5264782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 83, in sdpa_attention_forward 2025-11-03T16:38:19.5264916Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:19.5264919Z 2025-11-03T16:38:19.5265015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5265223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5265281Z res = mod(**inputs) 2025-11-03T16:38:19.5265529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5265605Z outputs = self.model.decoder( 2025-11-03T16:38:19.5265850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5265925Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5266133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5266208Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5266440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5266518Z return func(*args, **kwargs) 2025-11-03T16:38:19.5266769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-11-03T16:38:19.5266859Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:38:19.5267093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5267157Z return func(*args, **kwargs) 2025-11-03T16:38:19.5267401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-11-03T16:38:19.5267485Z attn_output = self.out_proj(attn_output) 2025-11-03T16:38:19.5267488Z 2025-11-03T16:38:19.5267584Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5267789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5267849Z res = mod(**inputs) 2025-11-03T16:38:19.5268099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5268174Z outputs = self.model.decoder( 2025-11-03T16:38:19.5268420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5268498Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5268707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5268787Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5269011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5269092Z return func(*args, **kwargs) 2025-11-03T16:38:19.5269346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5269460Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5269464Z 2025-11-03T16:38:19.5269568Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5269751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5269810Z res = mod(**inputs) 2025-11-03T16:38:19.5270064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5270133Z outputs = self.model.decoder( 2025-11-03T16:38:19.5270439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5270535Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5270791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5270889Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5271377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5271478Z return func(*args, **kwargs) 2025-11-03T16:38:19.5271804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-11-03T16:38:19.5271944Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:38:19.5272173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:19.5272286Z return self.act(input) 2025-11-03T16:38:19.5272289Z 2025-11-03T16:38:19.5272444Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5272696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5272779Z res = mod(**inputs) 2025-11-03T16:38:19.5273115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1635, in forward 2025-11-03T16:38:19.5273207Z outputs = self.model.decoder( 2025-11-03T16:38:19.5273489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1122, in forward 2025-11-03T16:38:19.5273641Z layer_outputs = decoder_layer( 2025-11-03T16:38:19.5273874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:19.5274001Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:19.5274325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:19.5274433Z return func(*args, **kwargs) 2025-11-03T16:38:19.5274751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-11-03T16:38:19.5274867Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:38:19.5274871Z 2025-11-03T16:38:19.5275032Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5275245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5275361Z res = mod(**inputs) 2025-11-03T16:38:19.5275630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1651, in forward 2025-11-03T16:38:19.5275745Z logits = self.lm_head(outputs[0]) 2025-11-03T16:38:19.5275819Z 2025-11-03T16:38:19.5275943Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:19.5276167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:19.5276298Z res = mod(**inputs) 2025-11-03T16:38:19.5276591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1657, in forward 2025-11-03T16:38:19.5276787Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:38:19.5276791Z 2025-11-03T16:38:28.6076093Z Compilation time (from dynamo_timed): 15.337183259 2025-11-03T16:38:28.6111841Z pass 2025-11-03T16:38:28.6112286Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:28.6113549Z TIMING: _recursive_pre_grad_passes:0.00721 _recursive_joint_graph_passes:0.61851 _recursive_post_grad_passes:0.06106 async_compile.wait:0.75525 code_gen:8.75922 inductor_compile:10.01351 backend_compile:12.9602 gc:0.00203 entire_frame_compile:15.33718 total_wall_time:15.33718 2025-11-03T16:38:28.6114789Z STATS: call_* op count: 333 | FakeTensorMode.__torch_dispatch__:7244 | FakeTensor.__torch_dispatch__:4382 | ProxyTorchDispatchMode.__torch_dispatch__:2006 2025-11-03T16:38:28.6115397Z Dynamo produced 1 graphs covering 333 ops with 0 graph breaks (0 unique) 2025-11-03T16:38:30.8699061Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:38:30.8700359Z import pynvml # type: ignore[import] 2025-11-03T16:38:34.0890143Z 2025-11-03T16:38:34.0995966Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-11-03T16:38:34.0998061Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-11-03T16:38:35.1064586Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-11-03T16:38:35.1065595Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-11-03T16:38:35.1067124Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-11-03T16:38:35.1068157Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-11-03T16:38:35.2780259Z 2025-11-03T16:38:35.2780980Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:38:35.2793078Z cpu eval RobertaForCausalLM 2025-11-03T16:38:35.8826383Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:36.1614800Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:36.4452476Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:43.4556226Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4556956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4557405Z res = mod(**inputs) 2025-11-03T16:38:43.4557873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4558389Z outputs = self.roberta( 2025-11-03T16:38:43.4558872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 797, in forward 2025-11-03T16:38:43.4559401Z embedding_output = self.embeddings( 2025-11-03T16:38:43.4560217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 88, in forward 2025-11-03T16:38:43.4560871Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-11-03T16:38:43.4561617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1547, in create_position_ids_from_input_ids 2025-11-03T16:38:43.4562195Z mask = input_ids.ne(padding_idx).int() 2025-11-03T16:38:43.4562373Z 2025-11-03T16:38:43.4562530Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4562867Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4563613Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4563885Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4564352Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4564644Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4564978Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4565227Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4565508Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4565841Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4566198Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4566430Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4566797Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4567819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4568297Z res = mod(**inputs) 2025-11-03T16:38:43.4568742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4579043Z outputs = self.roberta( 2025-11-03T16:38:43.4579639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 797, in forward 2025-11-03T16:38:43.4580108Z embedding_output = self.embeddings( 2025-11-03T16:38:43.4580562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 88, in forward 2025-11-03T16:38:43.4581295Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-11-03T16:38:43.4581941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-11-03T16:38:43.4582569Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-11-03T16:38:43.4582838Z 2025-11-03T16:38:43.4582965Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4583365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4583776Z res = mod(**inputs) 2025-11-03T16:38:43.4584245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4584671Z outputs = self.roberta( 2025-11-03T16:38:43.4585098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 797, in forward 2025-11-03T16:38:43.4585534Z embedding_output = self.embeddings( 2025-11-03T16:38:43.4585973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 88, in forward 2025-11-03T16:38:43.4586531Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-11-03T16:38:43.4587161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-11-03T16:38:43.4587773Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-11-03T16:38:43.4588024Z 2025-11-03T16:38:43.4588183Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4588572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4588923Z res = mod(**inputs) 2025-11-03T16:38:43.4589329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4589761Z outputs = self.roberta( 2025-11-03T16:38:43.4590138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4590535Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4590948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4591348Z layer_outputs = layer_module( 2025-11-03T16:38:43.4591711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4592095Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4592489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4592917Z return func(*args, **kwargs) 2025-11-03T16:38:43.4593302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4593720Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4594204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4594600Z return func(*args, **kwargs) 2025-11-03T16:38:43.4594997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4595414Z self_outputs = self.self( 2025-11-03T16:38:43.4595787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4596155Z return func(*args, **kwargs) 2025-11-03T16:38:43.4596560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4597088Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4597352Z 2025-11-03T16:38:43.4597465Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4597820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4598126Z res = mod(**inputs) 2025-11-03T16:38:43.4598501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4598889Z outputs = self.roberta( 2025-11-03T16:38:43.4599283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4599669Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4600057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4600450Z layer_outputs = layer_module( 2025-11-03T16:38:43.4600801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4601165Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4601536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4601907Z return func(*args, **kwargs) 2025-11-03T16:38:43.4602268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4602676Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4603042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4603396Z return func(*args, **kwargs) 2025-11-03T16:38:43.4603765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4604137Z self_outputs = self.self( 2025-11-03T16:38:43.4604484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4604828Z return func(*args, **kwargs) 2025-11-03T16:38:43.4605190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4605561Z self.key(current_states) 2025-11-03T16:38:43.4605671Z 2025-11-03T16:38:43.4605783Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4606134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4606439Z res = mod(**inputs) 2025-11-03T16:38:43.4606812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4607183Z outputs = self.roberta( 2025-11-03T16:38:43.4607557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4607937Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4608306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4608686Z layer_outputs = layer_module( 2025-11-03T16:38:43.4609033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4609385Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4609741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4610117Z return func(*args, **kwargs) 2025-11-03T16:38:43.4610481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4610870Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4611241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4611596Z return func(*args, **kwargs) 2025-11-03T16:38:43.4611961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4612338Z self_outputs = self.self( 2025-11-03T16:38:43.4612707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4613057Z return func(*args, **kwargs) 2025-11-03T16:38:43.4613754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4614140Z self.value(current_states) 2025-11-03T16:38:43.4614255Z 2025-11-03T16:38:43.4614345Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4614583Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4614924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4615236Z res = mod(**inputs) 2025-11-03T16:38:43.4615606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4615975Z outputs = self.roberta( 2025-11-03T16:38:43.4616358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4616730Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4617094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4617463Z layer_outputs = layer_module( 2025-11-03T16:38:43.4617789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4618118Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4618473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4618822Z return func(*args, **kwargs) 2025-11-03T16:38:43.4619175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4619553Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4619909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4620287Z return func(*args, **kwargs) 2025-11-03T16:38:43.4620646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4621013Z self_outputs = self.self( 2025-11-03T16:38:43.4621348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4621693Z return func(*args, **kwargs) 2025-11-03T16:38:43.4622048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4622475Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4622649Z 2025-11-03T16:38:43.4622757Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4623090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4623395Z res = mod(**inputs) 2025-11-03T16:38:43.4623771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4624135Z outputs = self.roberta( 2025-11-03T16:38:43.4624488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4624850Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4625214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4625579Z layer_outputs = layer_module( 2025-11-03T16:38:43.4625905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4626239Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4626622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4626972Z return func(*args, **kwargs) 2025-11-03T16:38:43.4627326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4627703Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4628054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4628400Z return func(*args, **kwargs) 2025-11-03T16:38:43.4628754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4629170Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4629606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4629975Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4630120Z 2025-11-03T16:38:43.4630218Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4630553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4630850Z res = mod(**inputs) 2025-11-03T16:38:43.4631188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4631552Z outputs = self.roberta( 2025-11-03T16:38:43.4631901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4632269Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4632630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4632994Z layer_outputs = layer_module( 2025-11-03T16:38:43.4633325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4633694Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4634170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4634575Z return func(*args, **kwargs) 2025-11-03T16:38:43.4634980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4635427Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4635812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4636227Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4636702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4637242Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4637733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4638166Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4638315Z 2025-11-03T16:38:43.4638428Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4638811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4639163Z res = mod(**inputs) 2025-11-03T16:38:43.4639559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4639970Z outputs = self.roberta( 2025-11-03T16:38:43.4640381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4640808Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4641228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4641647Z layer_outputs = layer_module( 2025-11-03T16:38:43.4642022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4642353Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4642718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4643068Z return func(*args, **kwargs) 2025-11-03T16:38:43.4643422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4643830Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4644199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4644572Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4644975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4645420Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4645829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4646229Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4646590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4646914Z return self.act(input) 2025-11-03T16:38:43.4647021Z 2025-11-03T16:38:43.4647127Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4647466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4647778Z res = mod(**inputs) 2025-11-03T16:38:43.4648122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4648485Z outputs = self.roberta( 2025-11-03T16:38:43.4648838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4649198Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4649568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4649930Z layer_outputs = layer_module( 2025-11-03T16:38:43.4650255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4650594Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4650954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4651304Z return func(*args, **kwargs) 2025-11-03T16:38:43.4651657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4652033Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4652407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4652767Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4653162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4653629Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4654051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4654434Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4654563Z 2025-11-03T16:38:43.4654659Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4654996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4655297Z res = mod(**inputs) 2025-11-03T16:38:43.4655640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4655996Z outputs = self.roberta( 2025-11-03T16:38:43.4656346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4656725Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4657094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4657464Z layer_outputs = layer_module( 2025-11-03T16:38:43.4657782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4658121Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4658475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4658820Z return func(*args, **kwargs) 2025-11-03T16:38:43.4659168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4659541Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4659898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4660244Z return func(*args, **kwargs) 2025-11-03T16:38:43.4660598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4660971Z self_outputs = self.self( 2025-11-03T16:38:43.4661311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4661657Z return func(*args, **kwargs) 2025-11-03T16:38:43.4662009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4662509Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4662753Z 2025-11-03T16:38:43.4662849Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4663187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4663488Z res = mod(**inputs) 2025-11-03T16:38:43.4663831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4664225Z outputs = self.roberta( 2025-11-03T16:38:43.4664576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4664957Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4665334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4665723Z layer_outputs = layer_module( 2025-11-03T16:38:43.4666052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4666403Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4666785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4667150Z return func(*args, **kwargs) 2025-11-03T16:38:43.4667528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4667918Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4668296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4668660Z return func(*args, **kwargs) 2025-11-03T16:38:43.4669033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4669417Z self_outputs = self.self( 2025-11-03T16:38:43.4669768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4670150Z return func(*args, **kwargs) 2025-11-03T16:38:43.4670519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4670899Z self.key(current_states) 2025-11-03T16:38:43.4671010Z 2025-11-03T16:38:43.4671111Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4671458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4671767Z res = mod(**inputs) 2025-11-03T16:38:43.4672118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4672494Z outputs = self.roberta( 2025-11-03T16:38:43.4672847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4673225Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4673600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4673974Z layer_outputs = layer_module( 2025-11-03T16:38:43.4674401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4674770Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4675157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4675534Z return func(*args, **kwargs) 2025-11-03T16:38:43.4675918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4676327Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4676707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4677079Z return func(*args, **kwargs) 2025-11-03T16:38:43.4677454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4677863Z self_outputs = self.self( 2025-11-03T16:38:43.4678217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4678588Z return func(*args, **kwargs) 2025-11-03T16:38:43.4678970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4679366Z self.value(current_states) 2025-11-03T16:38:43.4679486Z 2025-11-03T16:38:43.4679569Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4679814Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4680175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4680497Z res = mod(**inputs) 2025-11-03T16:38:43.4680887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4681271Z outputs = self.roberta( 2025-11-03T16:38:43.4681637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4682064Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4682453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4682837Z layer_outputs = layer_module( 2025-11-03T16:38:43.4683205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4683563Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4683986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4684398Z return func(*args, **kwargs) 2025-11-03T16:38:43.4684789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4685215Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4685633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4686016Z return func(*args, **kwargs) 2025-11-03T16:38:43.4686421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4686815Z self_outputs = self.self( 2025-11-03T16:38:43.4687189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4687570Z return func(*args, **kwargs) 2025-11-03T16:38:43.4687964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4688424Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4688648Z 2025-11-03T16:38:43.4688754Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4689115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4689460Z res = mod(**inputs) 2025-11-03T16:38:43.4689845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4690250Z outputs = self.roberta( 2025-11-03T16:38:43.4690635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4691025Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4691415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4691816Z layer_outputs = layer_module( 2025-11-03T16:38:43.4692177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4692537Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4692933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4693281Z return func(*args, **kwargs) 2025-11-03T16:38:43.4693629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4694010Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4694367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4694718Z return func(*args, **kwargs) 2025-11-03T16:38:43.4695089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4695507Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4695931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4696311Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4696441Z 2025-11-03T16:38:43.4696547Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4696884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4697179Z res = mod(**inputs) 2025-11-03T16:38:43.4697524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4697889Z outputs = self.roberta( 2025-11-03T16:38:43.4698257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4698618Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4698976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4699339Z layer_outputs = layer_module( 2025-11-03T16:38:43.4699664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4700003Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4700346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4700690Z return func(*args, **kwargs) 2025-11-03T16:38:43.4701042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4701419Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4701791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4702172Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4702576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4703028Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4703446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4703846Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4703981Z 2025-11-03T16:38:43.4704082Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4704435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4704749Z res = mod(**inputs) 2025-11-03T16:38:43.4705108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4705482Z outputs = self.roberta( 2025-11-03T16:38:43.4705832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4706197Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4706560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4706926Z layer_outputs = layer_module( 2025-11-03T16:38:43.4707243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4707581Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4707945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4708290Z return func(*args, **kwargs) 2025-11-03T16:38:43.4708644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4709020Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4709411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4709786Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4710202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4710649Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4711085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4711507Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4711879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4712214Z return self.act(input) 2025-11-03T16:38:43.4712321Z 2025-11-03T16:38:43.4712422Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4712771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4713081Z res = mod(**inputs) 2025-11-03T16:38:43.4713585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4713983Z outputs = self.roberta( 2025-11-03T16:38:43.4714407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4714813Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4715211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4715648Z layer_outputs = layer_module( 2025-11-03T16:38:43.4715974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4716323Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4716691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4717047Z return func(*args, **kwargs) 2025-11-03T16:38:43.4717411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4717791Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4718180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4718555Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4718962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4719454Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4719880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4720270Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4720408Z 2025-11-03T16:38:43.4720508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4720851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4721150Z res = mod(**inputs) 2025-11-03T16:38:43.4721528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4721901Z outputs = self.roberta( 2025-11-03T16:38:43.4722263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4722643Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4723005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4723378Z layer_outputs = layer_module( 2025-11-03T16:38:43.4723710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4724062Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4724426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4724777Z return func(*args, **kwargs) 2025-11-03T16:38:43.4725166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4725561Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4725922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4726262Z return func(*args, **kwargs) 2025-11-03T16:38:43.4726618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4726980Z self_outputs = self.self( 2025-11-03T16:38:43.4727320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4727664Z return func(*args, **kwargs) 2025-11-03T16:38:43.4728008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4728510Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4728760Z 2025-11-03T16:38:43.4728875Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4729216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4729514Z res = mod(**inputs) 2025-11-03T16:38:43.4729849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4730212Z outputs = self.roberta( 2025-11-03T16:38:43.4730562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4730927Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4731279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4731640Z layer_outputs = layer_module( 2025-11-03T16:38:43.4731962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4732315Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4732674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4733011Z return func(*args, **kwargs) 2025-11-03T16:38:43.4733364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4733738Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4734093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4734437Z return func(*args, **kwargs) 2025-11-03T16:38:43.4734830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4735199Z self_outputs = self.self( 2025-11-03T16:38:43.4735535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4735880Z return func(*args, **kwargs) 2025-11-03T16:38:43.4736226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4736585Z self.key(current_states) 2025-11-03T16:38:43.4736698Z 2025-11-03T16:38:43.4736794Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4737132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4737432Z res = mod(**inputs) 2025-11-03T16:38:43.4737782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4738152Z outputs = self.roberta( 2025-11-03T16:38:43.4738504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4738878Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4739242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4739609Z layer_outputs = layer_module( 2025-11-03T16:38:43.4739934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4740276Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4740633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4740977Z return func(*args, **kwargs) 2025-11-03T16:38:43.4741338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4741717Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4742101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4742444Z return func(*args, **kwargs) 2025-11-03T16:38:43.4742795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4743158Z self_outputs = self.self( 2025-11-03T16:38:43.4743490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4743831Z return func(*args, **kwargs) 2025-11-03T16:38:43.4744176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4744543Z self.value(current_states) 2025-11-03T16:38:43.4744658Z 2025-11-03T16:38:43.4744738Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4744963Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4745313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4745607Z res = mod(**inputs) 2025-11-03T16:38:43.4745947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4746310Z outputs = self.roberta( 2025-11-03T16:38:43.4746662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4747032Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4747388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4747756Z layer_outputs = layer_module( 2025-11-03T16:38:43.4748098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4748440Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4748795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4749139Z return func(*args, **kwargs) 2025-11-03T16:38:43.4749491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4749864Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4750221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4750557Z return func(*args, **kwargs) 2025-11-03T16:38:43.4750923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4751293Z self_outputs = self.self( 2025-11-03T16:38:43.4751631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4751975Z return func(*args, **kwargs) 2025-11-03T16:38:43.4752330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4752755Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4752936Z 2025-11-03T16:38:43.4753035Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4753376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4753669Z res = mod(**inputs) 2025-11-03T16:38:43.4754015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4754478Z outputs = self.roberta( 2025-11-03T16:38:43.4754853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4755270Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4755656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4756037Z layer_outputs = layer_module( 2025-11-03T16:38:43.4756377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4756729Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4757096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4757464Z return func(*args, **kwargs) 2025-11-03T16:38:43.4757838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4758238Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4758630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4758991Z return func(*args, **kwargs) 2025-11-03T16:38:43.4759362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4759806Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4760246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4760644Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4760780Z 2025-11-03T16:38:43.4760880Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4761253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4761570Z res = mod(**inputs) 2025-11-03T16:38:43.4761929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4762310Z outputs = self.roberta( 2025-11-03T16:38:43.4762669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4763057Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4763435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4763818Z layer_outputs = layer_module( 2025-11-03T16:38:43.4764152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4764505Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4764893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4765261Z return func(*args, **kwargs) 2025-11-03T16:38:43.4765635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4766022Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4766418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4766800Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4767204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4767652Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4768063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4768447Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4768586Z 2025-11-03T16:38:43.4768701Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4769041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4769341Z res = mod(**inputs) 2025-11-03T16:38:43.4769695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4770069Z outputs = self.roberta( 2025-11-03T16:38:43.4770426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4770801Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4771170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4771548Z layer_outputs = layer_module( 2025-11-03T16:38:43.4771881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4772244Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4772611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4772964Z return func(*args, **kwargs) 2025-11-03T16:38:43.4773329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4773718Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4774103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4774470Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4774896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4775347Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4775773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4776181Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4776541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4776868Z return self.act(input) 2025-11-03T16:38:43.4776980Z 2025-11-03T16:38:43.4777079Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4777427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4777737Z res = mod(**inputs) 2025-11-03T16:38:43.4778100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4778480Z outputs = self.roberta( 2025-11-03T16:38:43.4778837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4779215Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4779580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4779953Z layer_outputs = layer_module( 2025-11-03T16:38:43.4780285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4780637Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4781002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4781349Z return func(*args, **kwargs) 2025-11-03T16:38:43.4781715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4782101Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4782504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4782893Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4783299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4783779Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4784207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4784588Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4784717Z 2025-11-03T16:38:43.4784825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4785161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4785485Z res = mod(**inputs) 2025-11-03T16:38:43.4785834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4786205Z outputs = self.roberta( 2025-11-03T16:38:43.4786556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4786929Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4787302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4787676Z layer_outputs = layer_module( 2025-11-03T16:38:43.4788008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4788361Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4788733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4789099Z return func(*args, **kwargs) 2025-11-03T16:38:43.4789471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4789855Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4790232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4790592Z return func(*args, **kwargs) 2025-11-03T16:38:43.4790967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4791347Z self_outputs = self.self( 2025-11-03T16:38:43.4791724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4792084Z return func(*args, **kwargs) 2025-11-03T16:38:43.4792452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4792966Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4793217Z 2025-11-03T16:38:43.4793576Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4793916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4794315Z res = mod(**inputs) 2025-11-03T16:38:43.4794680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4795060Z outputs = self.roberta( 2025-11-03T16:38:43.4795431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4795806Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4796205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4796579Z layer_outputs = layer_module( 2025-11-03T16:38:43.4796910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4797251Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4797619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4797977Z return func(*args, **kwargs) 2025-11-03T16:38:43.4798348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4798722Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4799072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4799434Z return func(*args, **kwargs) 2025-11-03T16:38:43.4799787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4800148Z self_outputs = self.self( 2025-11-03T16:38:43.4800480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4800814Z return func(*args, **kwargs) 2025-11-03T16:38:43.4801165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4801529Z self.key(current_states) 2025-11-03T16:38:43.4801634Z 2025-11-03T16:38:43.4801738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4802081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4802384Z res = mod(**inputs) 2025-11-03T16:38:43.4802733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4803100Z outputs = self.roberta( 2025-11-03T16:38:43.4803455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4803822Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4804191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4804562Z layer_outputs = layer_module( 2025-11-03T16:38:43.4804892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4805246Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4805595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4805950Z return func(*args, **kwargs) 2025-11-03T16:38:43.4806304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4806685Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4807036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4807385Z return func(*args, **kwargs) 2025-11-03T16:38:43.4807738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4808105Z self_outputs = self.self( 2025-11-03T16:38:43.4808447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4808785Z return func(*args, **kwargs) 2025-11-03T16:38:43.4809137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4809523Z self.value(current_states) 2025-11-03T16:38:43.4809634Z 2025-11-03T16:38:43.4809718Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4809938Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4810275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4810577Z res = mod(**inputs) 2025-11-03T16:38:43.4810921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4811285Z outputs = self.roberta( 2025-11-03T16:38:43.4811630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4811999Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4812361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4812747Z layer_outputs = layer_module( 2025-11-03T16:38:43.4813069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4813554Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4813919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4814283Z return func(*args, **kwargs) 2025-11-03T16:38:43.4814645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4815021Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4815422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4815773Z return func(*args, **kwargs) 2025-11-03T16:38:43.4816127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4816487Z self_outputs = self.self( 2025-11-03T16:38:43.4816816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4817165Z return func(*args, **kwargs) 2025-11-03T16:38:43.4817521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4817946Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4818117Z 2025-11-03T16:38:43.4818223Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4818576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4818877Z res = mod(**inputs) 2025-11-03T16:38:43.4819228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4819595Z outputs = self.roberta( 2025-11-03T16:38:43.4819942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4820310Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4820678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4821050Z layer_outputs = layer_module( 2025-11-03T16:38:43.4821376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4821714Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4822073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4822451Z return func(*args, **kwargs) 2025-11-03T16:38:43.4822806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4823183Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4823529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4823876Z return func(*args, **kwargs) 2025-11-03T16:38:43.4824231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4824648Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4825059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4825436Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4825608Z 2025-11-03T16:38:43.4825704Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4826038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4826336Z res = mod(**inputs) 2025-11-03T16:38:43.4826667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4827028Z outputs = self.roberta( 2025-11-03T16:38:43.4827387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4827755Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4828117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4828491Z layer_outputs = layer_module( 2025-11-03T16:38:43.4828819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4829161Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4829516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4829859Z return func(*args, **kwargs) 2025-11-03T16:38:43.4830214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4830594Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4830971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4831347Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4831757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4832205Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4832616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4832987Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4833114Z 2025-11-03T16:38:43.4833216Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4833546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4833888Z res = mod(**inputs) 2025-11-03T16:38:43.4834306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4834699Z outputs = self.roberta( 2025-11-03T16:38:43.4835096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4835470Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4835857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4836231Z layer_outputs = layer_module( 2025-11-03T16:38:43.4836559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4836893Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4837254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4837605Z return func(*args, **kwargs) 2025-11-03T16:38:43.4837966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4838348Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4838720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4839103Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4839498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4839938Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4840339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4840738Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4841099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4841427Z return self.act(input) 2025-11-03T16:38:43.4841530Z 2025-11-03T16:38:43.4841666Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4841997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4842310Z res = mod(**inputs) 2025-11-03T16:38:43.4842660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4843037Z outputs = self.roberta( 2025-11-03T16:38:43.4843392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4843760Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4844130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4844505Z layer_outputs = layer_module( 2025-11-03T16:38:43.4844854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4845190Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4845542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4845893Z return func(*args, **kwargs) 2025-11-03T16:38:43.4846244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4846623Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4846989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4847354Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4847749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4848205Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4848630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4849020Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4849156Z 2025-11-03T16:38:43.4849252Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4849587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4849886Z res = mod(**inputs) 2025-11-03T16:38:43.4850228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4850583Z outputs = self.roberta( 2025-11-03T16:38:43.4850930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4851296Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4851660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4852036Z layer_outputs = layer_module( 2025-11-03T16:38:43.4852362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4852704Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4853062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4853412Z return func(*args, **kwargs) 2025-11-03T16:38:43.4853773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4854152Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4854516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4854883Z return func(*args, **kwargs) 2025-11-03T16:38:43.4855236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4855607Z self_outputs = self.self( 2025-11-03T16:38:43.4855945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4856291Z return func(*args, **kwargs) 2025-11-03T16:38:43.4856643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4857131Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4857383Z 2025-11-03T16:38:43.4857480Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4857850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4858156Z res = mod(**inputs) 2025-11-03T16:38:43.4858502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4858859Z outputs = self.roberta( 2025-11-03T16:38:43.4859210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4859576Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4859942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4860306Z layer_outputs = layer_module( 2025-11-03T16:38:43.4860633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4860973Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4861334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4861684Z return func(*args, **kwargs) 2025-11-03T16:38:43.4862029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4862424Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4862782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4863130Z return func(*args, **kwargs) 2025-11-03T16:38:43.4863484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4863842Z self_outputs = self.self( 2025-11-03T16:38:43.4864178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4864523Z return func(*args, **kwargs) 2025-11-03T16:38:43.4864881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4865259Z self.key(current_states) 2025-11-03T16:38:43.4865367Z 2025-11-03T16:38:43.4865464Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4865799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4866096Z res = mod(**inputs) 2025-11-03T16:38:43.4866437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4866792Z outputs = self.roberta( 2025-11-03T16:38:43.4867138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4867504Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4867894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4868267Z layer_outputs = layer_module( 2025-11-03T16:38:43.4868590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4868934Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4869292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4869640Z return func(*args, **kwargs) 2025-11-03T16:38:43.4869988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4870368Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4870728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4871096Z return func(*args, **kwargs) 2025-11-03T16:38:43.4871461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4871823Z self_outputs = self.self( 2025-11-03T16:38:43.4872167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4872517Z return func(*args, **kwargs) 2025-11-03T16:38:43.4872877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4873248Z self.value(current_states) 2025-11-03T16:38:43.4873358Z 2025-11-03T16:38:43.4873435Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4873665Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4874014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4874407Z res = mod(**inputs) 2025-11-03T16:38:43.4874771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4875180Z outputs = self.roberta( 2025-11-03T16:38:43.4875554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4875962Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4876347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4876746Z layer_outputs = layer_module( 2025-11-03T16:38:43.4877091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4877453Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4877829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4878189Z return func(*args, **kwargs) 2025-11-03T16:38:43.4878561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4878976Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4879354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4879718Z return func(*args, **kwargs) 2025-11-03T16:38:43.4880082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4880464Z self_outputs = self.self( 2025-11-03T16:38:43.4880818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4881177Z return func(*args, **kwargs) 2025-11-03T16:38:43.4881563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4882000Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4882192Z 2025-11-03T16:38:43.4882294Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4882654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4882970Z res = mod(**inputs) 2025-11-03T16:38:43.4883323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4883709Z outputs = self.roberta( 2025-11-03T16:38:43.4884082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4884467Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4884866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4885246Z layer_outputs = layer_module( 2025-11-03T16:38:43.4885592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4885959Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4886348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4886721Z return func(*args, **kwargs) 2025-11-03T16:38:43.4887094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4887497Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4887881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4888255Z return func(*args, **kwargs) 2025-11-03T16:38:43.4888631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4889096Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4889535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4889934Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4890071Z 2025-11-03T16:38:43.4890182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4890530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4890850Z res = mod(**inputs) 2025-11-03T16:38:43.4891213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4891581Z outputs = self.roberta( 2025-11-03T16:38:43.4891937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4892330Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4892698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4893064Z layer_outputs = layer_module( 2025-11-03T16:38:43.4893393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4893727Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4894085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4894437Z return func(*args, **kwargs) 2025-11-03T16:38:43.4894797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4895195Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4895565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4895936Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4896335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4896777Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4897187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4897554Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4897690Z 2025-11-03T16:38:43.4897787Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4898139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4898440Z res = mod(**inputs) 2025-11-03T16:38:43.4898780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4899150Z outputs = self.roberta( 2025-11-03T16:38:43.4899498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4899864Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4900220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4900581Z layer_outputs = layer_module( 2025-11-03T16:38:43.4900907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4901246Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4901603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4901950Z return func(*args, **kwargs) 2025-11-03T16:38:43.4902298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4902699Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4903077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4903450Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4903846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4904290Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4904712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4905121Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4905487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4905820Z return self.act(input) 2025-11-03T16:38:43.4905932Z 2025-11-03T16:38:43.4906028Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4906364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4906665Z res = mod(**inputs) 2025-11-03T16:38:43.4907012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4907367Z outputs = self.roberta( 2025-11-03T16:38:43.4907716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4908083Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4908459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4908831Z layer_outputs = layer_module( 2025-11-03T16:38:43.4909150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4909487Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4909841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4910190Z return func(*args, **kwargs) 2025-11-03T16:38:43.4910536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4910913Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4911296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4911666Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4912065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4912512Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4912934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4913502Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4913638Z 2025-11-03T16:38:43.4913747Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4914144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4914462Z res = mod(**inputs) 2025-11-03T16:38:43.4914838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4915216Z outputs = self.roberta( 2025-11-03T16:38:43.4915580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4916039Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4916415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4916798Z layer_outputs = layer_module( 2025-11-03T16:38:43.4917144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4917504Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4917869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4918234Z return func(*args, **kwargs) 2025-11-03T16:38:43.4918613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4919010Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4919419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4919779Z return func(*args, **kwargs) 2025-11-03T16:38:43.4920153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4920537Z self_outputs = self.self( 2025-11-03T16:38:43.4920895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4921253Z return func(*args, **kwargs) 2025-11-03T16:38:43.4921625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4922172Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4922445Z 2025-11-03T16:38:43.4922551Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4922909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4923220Z res = mod(**inputs) 2025-11-03T16:38:43.4923583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4923968Z outputs = self.roberta( 2025-11-03T16:38:43.4924337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4924722Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4925078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4925473Z layer_outputs = layer_module( 2025-11-03T16:38:43.4925804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4926149Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4926499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4926850Z return func(*args, **kwargs) 2025-11-03T16:38:43.4927210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4927588Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4927947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4928293Z return func(*args, **kwargs) 2025-11-03T16:38:43.4928654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4929023Z self_outputs = self.self( 2025-11-03T16:38:43.4929363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4929730Z return func(*args, **kwargs) 2025-11-03T16:38:43.4930072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4930435Z self.key(current_states) 2025-11-03T16:38:43.4930547Z 2025-11-03T16:38:43.4930644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4930977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4931270Z res = mod(**inputs) 2025-11-03T16:38:43.4931609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4931970Z outputs = self.roberta( 2025-11-03T16:38:43.4932316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4932705Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4933064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4933433Z layer_outputs = layer_module( 2025-11-03T16:38:43.4933759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4934101Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4934462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4934802Z return func(*args, **kwargs) 2025-11-03T16:38:43.4935847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4936240Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4936607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4936950Z return func(*args, **kwargs) 2025-11-03T16:38:43.4937305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4937673Z self_outputs = self.self( 2025-11-03T16:38:43.4938011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4938364Z return func(*args, **kwargs) 2025-11-03T16:38:43.4938710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4939074Z self.value(current_states) 2025-11-03T16:38:43.4939207Z 2025-11-03T16:38:43.4939287Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4939513Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4939846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4940145Z res = mod(**inputs) 2025-11-03T16:38:43.4940485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4940849Z outputs = self.roberta( 2025-11-03T16:38:43.4941201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4941555Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4941919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4942283Z layer_outputs = layer_module( 2025-11-03T16:38:43.4942610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4942947Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4943324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4943671Z return func(*args, **kwargs) 2025-11-03T16:38:43.4944026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4944404Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4944629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4944692Z return func(*args, **kwargs) 2025-11-03T16:38:43.4944941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4945008Z self_outputs = self.self( 2025-11-03T16:38:43.4945236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4945348Z return func(*args, **kwargs) 2025-11-03T16:38:43.4945602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4945727Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4945730Z 2025-11-03T16:38:43.4945827Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4946023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4946083Z res = mod(**inputs) 2025-11-03T16:38:43.4946339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4946405Z outputs = self.roberta( 2025-11-03T16:38:43.4946672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4946750Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4946993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4947066Z layer_outputs = layer_module( 2025-11-03T16:38:43.4947275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4947354Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4947578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4947642Z return func(*args, **kwargs) 2025-11-03T16:38:43.4947915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4947995Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4948236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4948303Z return func(*args, **kwargs) 2025-11-03T16:38:43.4948551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4948681Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4948931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4949019Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4949022Z 2025-11-03T16:38:43.4949124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4949321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4949383Z res = mod(**inputs) 2025-11-03T16:38:43.4949631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4949723Z outputs = self.roberta( 2025-11-03T16:38:43.4949971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4950047Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4950295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4950364Z layer_outputs = layer_module( 2025-11-03T16:38:43.4950586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4950659Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4950901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4950966Z return func(*args, **kwargs) 2025-11-03T16:38:43.4951231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4951321Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4951567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4951648Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4951928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4952050Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4952301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4952396Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4952400Z 2025-11-03T16:38:43.4952509Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4952699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4952767Z res = mod(**inputs) 2025-11-03T16:38:43.4953016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4953080Z outputs = self.roberta( 2025-11-03T16:38:43.4953337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4953405Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4953658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4953744Z layer_outputs = layer_module( 2025-11-03T16:38:43.4953966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4954044Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4954346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4954426Z return func(*args, **kwargs) 2025-11-03T16:38:43.4954734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4954827Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4955077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4955154Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4955453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4955575Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4955909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4956020Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4956239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4956307Z return self.act(input) 2025-11-03T16:38:43.4956311Z 2025-11-03T16:38:43.4956413Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4956611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4956674Z res = mod(**inputs) 2025-11-03T16:38:43.4956938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4957004Z outputs = self.roberta( 2025-11-03T16:38:43.4957252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4957351Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4957599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4957675Z layer_outputs = layer_module( 2025-11-03T16:38:43.4957891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4957966Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4958207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4958277Z return func(*args, **kwargs) 2025-11-03T16:38:43.4958549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4958631Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4958884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4958956Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4959233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4959369Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4959614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4959701Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4959704Z 2025-11-03T16:38:43.4959816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4960006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4960076Z res = mod(**inputs) 2025-11-03T16:38:43.4960331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4960403Z outputs = self.roberta( 2025-11-03T16:38:43.4960648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4960722Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4960967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4961035Z layer_outputs = layer_module( 2025-11-03T16:38:43.4961255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4961330Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4961569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4961650Z return func(*args, **kwargs) 2025-11-03T16:38:43.4961896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4961984Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4962212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4962283Z return func(*args, **kwargs) 2025-11-03T16:38:43.4962530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4962597Z self_outputs = self.self( 2025-11-03T16:38:43.4962840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4962903Z return func(*args, **kwargs) 2025-11-03T16:38:43.4963190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4963391Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4963395Z 2025-11-03T16:38:43.4963502Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4963698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4963760Z res = mod(**inputs) 2025-11-03T16:38:43.4964017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4964082Z outputs = self.roberta( 2025-11-03T16:38:43.4964357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4964426Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4964677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4964753Z layer_outputs = layer_module( 2025-11-03T16:38:43.4964964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4965049Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4965286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4965358Z return func(*args, **kwargs) 2025-11-03T16:38:43.4965610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4965709Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4965946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4966012Z return func(*args, **kwargs) 2025-11-03T16:38:43.4966262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4966327Z self_outputs = self.self( 2025-11-03T16:38:43.4966553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4966623Z return func(*args, **kwargs) 2025-11-03T16:38:43.4966863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4966935Z self.key(current_states) 2025-11-03T16:38:43.4966938Z 2025-11-03T16:38:43.4967034Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4967226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4967286Z res = mod(**inputs) 2025-11-03T16:38:43.4967547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4967619Z outputs = self.roberta( 2025-11-03T16:38:43.4967859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4967934Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4968174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4968240Z layer_outputs = layer_module( 2025-11-03T16:38:43.4968450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4968524Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4968754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4968835Z return func(*args, **kwargs) 2025-11-03T16:38:43.4969073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4969157Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4969388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4969459Z return func(*args, **kwargs) 2025-11-03T16:38:43.4969704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4969782Z self_outputs = self.self( 2025-11-03T16:38:43.4970014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4970092Z return func(*args, **kwargs) 2025-11-03T16:38:43.4970349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4970419Z self.value(current_states) 2025-11-03T16:38:43.4970423Z 2025-11-03T16:38:43.4970507Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4970603Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4970788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4970857Z res = mod(**inputs) 2025-11-03T16:38:43.4971102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4971174Z outputs = self.roberta( 2025-11-03T16:38:43.4971433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4971505Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4971759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4971828Z layer_outputs = layer_module( 2025-11-03T16:38:43.4972045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4972120Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4972360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4972425Z return func(*args, **kwargs) 2025-11-03T16:38:43.4972672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4972756Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4972994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4973069Z return func(*args, **kwargs) 2025-11-03T16:38:43.4973333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4973401Z self_outputs = self.self( 2025-11-03T16:38:43.4973649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4973713Z return func(*args, **kwargs) 2025-11-03T16:38:43.4973968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.4974094Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.4974098Z 2025-11-03T16:38:43.4974201Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4974391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4974452Z res = mod(**inputs) 2025-11-03T16:38:43.4974727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4974793Z outputs = self.roberta( 2025-11-03T16:38:43.4975049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4975118Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4975366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4975440Z layer_outputs = layer_module( 2025-11-03T16:38:43.4975651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4975735Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4975981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4976050Z return func(*args, **kwargs) 2025-11-03T16:38:43.4976304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4976381Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4976617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4976682Z return func(*args, **kwargs) 2025-11-03T16:38:43.4976932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.4977054Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.4977315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.4977405Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4977409Z 2025-11-03T16:38:43.4977508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4977702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4977764Z res = mod(**inputs) 2025-11-03T16:38:43.4978015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4978088Z outputs = self.roberta( 2025-11-03T16:38:43.4978336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4978412Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4978663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4978732Z layer_outputs = layer_module( 2025-11-03T16:38:43.4978953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4979045Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4979283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4979348Z return func(*args, **kwargs) 2025-11-03T16:38:43.4979602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4979681Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4979925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4980005Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4980284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4980423Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4980672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.4980751Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4980762Z 2025-11-03T16:38:43.4980860Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4981046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4981117Z res = mod(**inputs) 2025-11-03T16:38:43.4981365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4981437Z outputs = self.roberta( 2025-11-03T16:38:43.4981697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4981766Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4982021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4982088Z layer_outputs = layer_module( 2025-11-03T16:38:43.4982310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4982383Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4982613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4982691Z return func(*args, **kwargs) 2025-11-03T16:38:43.4982939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4983040Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4983299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4983382Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4983672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.4983790Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.4984057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.4984168Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.4984396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.4984465Z return self.act(input) 2025-11-03T16:38:43.4984468Z 2025-11-03T16:38:43.4984569Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4984764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4984848Z res = mod(**inputs) 2025-11-03T16:38:43.4985103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4985168Z outputs = self.roberta( 2025-11-03T16:38:43.4985417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4985492Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4985742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4985817Z layer_outputs = layer_module( 2025-11-03T16:38:43.4986032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4986115Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4986346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4986429Z return func(*args, **kwargs) 2025-11-03T16:38:43.4986686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.4986764Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.4987019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.4987092Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.4987368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.4987503Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.4987783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.4987874Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.4987877Z 2025-11-03T16:38:43.4987974Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4988169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4988232Z res = mod(**inputs) 2025-11-03T16:38:43.4988480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4988553Z outputs = self.roberta( 2025-11-03T16:38:43.4988799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4988877Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4989141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4989211Z layer_outputs = layer_module( 2025-11-03T16:38:43.4989430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4989505Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4989739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4989804Z return func(*args, **kwargs) 2025-11-03T16:38:43.4990058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4990136Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4990365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4990439Z return func(*args, **kwargs) 2025-11-03T16:38:43.4990685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4990775Z self_outputs = self.self( 2025-11-03T16:38:43.4991003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4991067Z return func(*args, **kwargs) 2025-11-03T16:38:43.4991319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.4991517Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.4991520Z 2025-11-03T16:38:43.4991627Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4991815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4991883Z res = mod(**inputs) 2025-11-03T16:38:43.4992130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4992209Z outputs = self.roberta( 2025-11-03T16:38:43.4992458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4992527Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4992778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4992845Z layer_outputs = layer_module( 2025-11-03T16:38:43.4993054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4993136Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4993381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4993456Z return func(*args, **kwargs) 2025-11-03T16:38:43.4993704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4993784Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4994021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4994147Z return func(*args, **kwargs) 2025-11-03T16:38:43.4994410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4994477Z self_outputs = self.self( 2025-11-03T16:38:43.4994715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4994800Z return func(*args, **kwargs) 2025-11-03T16:38:43.4995051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.4995131Z self.key(current_states) 2025-11-03T16:38:43.4995134Z 2025-11-03T16:38:43.4995233Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4995427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4995490Z res = mod(**inputs) 2025-11-03T16:38:43.4995738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4995810Z outputs = self.roberta( 2025-11-03T16:38:43.4996058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4996135Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.4996386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.4996454Z layer_outputs = layer_module( 2025-11-03T16:38:43.4996673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.4996767Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.4997004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4997069Z return func(*args, **kwargs) 2025-11-03T16:38:43.4997320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.4997399Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.4997629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4997703Z return func(*args, **kwargs) 2025-11-03T16:38:43.4997948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.4998043Z self_outputs = self.self( 2025-11-03T16:38:43.4998275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.4998341Z return func(*args, **kwargs) 2025-11-03T16:38:43.4998593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.4998660Z self.value(current_states) 2025-11-03T16:38:43.4998664Z 2025-11-03T16:38:43.4998750Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.4998848Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.4999035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.4999107Z res = mod(**inputs) 2025-11-03T16:38:43.4999369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.4999443Z outputs = self.roberta( 2025-11-03T16:38:43.4999693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.4999769Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5000018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5000084Z layer_outputs = layer_module( 2025-11-03T16:38:43.5000304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5000391Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5000636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5000703Z return func(*args, **kwargs) 2025-11-03T16:38:43.5000944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5001030Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5001255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5001325Z return func(*args, **kwargs) 2025-11-03T16:38:43.5001565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5001630Z self_outputs = self.self( 2025-11-03T16:38:43.5001861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5001924Z return func(*args, **kwargs) 2025-11-03T16:38:43.5002173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.5002297Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.5002317Z 2025-11-03T16:38:43.5002421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5002604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5002664Z res = mod(**inputs) 2025-11-03T16:38:43.5002911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5002974Z outputs = self.roberta( 2025-11-03T16:38:43.5003220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5003285Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5003529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5003601Z layer_outputs = layer_module( 2025-11-03T16:38:43.5003821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5003903Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5004128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5004199Z return func(*args, **kwargs) 2025-11-03T16:38:43.5004439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5004514Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5004748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5004811Z return func(*args, **kwargs) 2025-11-03T16:38:43.5005085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.5005206Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.5005451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.5005537Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5005540Z 2025-11-03T16:38:43.5005636Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5005825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5005885Z res = mod(**inputs) 2025-11-03T16:38:43.5006131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5006203Z outputs = self.roberta( 2025-11-03T16:38:43.5006461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5006538Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5006781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5006853Z layer_outputs = layer_module( 2025-11-03T16:38:43.5007059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5007131Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5007363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5007427Z return func(*args, **kwargs) 2025-11-03T16:38:43.5007676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5007757Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5007996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5008091Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5008362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5008482Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5008726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.5008810Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5008813Z 2025-11-03T16:38:43.5008909Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5009095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5009163Z res = mod(**inputs) 2025-11-03T16:38:43.5009413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5009512Z outputs = self.roberta( 2025-11-03T16:38:43.5009752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5009821Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5010068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5010133Z layer_outputs = layer_module( 2025-11-03T16:38:43.5010344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5010417Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5010663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5010727Z return func(*args, **kwargs) 2025-11-03T16:38:43.5010970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5011057Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5011296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5011373Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5011649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5011761Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5012026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.5012136Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.5012343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.5012411Z return self.act(input) 2025-11-03T16:38:43.5012415Z 2025-11-03T16:38:43.5012518Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5012701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5012760Z res = mod(**inputs) 2025-11-03T16:38:43.5013010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5013074Z outputs = self.roberta( 2025-11-03T16:38:43.5013483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5013557Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5013802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5013920Z layer_outputs = layer_module( 2025-11-03T16:38:43.5014127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5014211Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5014437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5014502Z return func(*args, **kwargs) 2025-11-03T16:38:43.5014752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5014829Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5015077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5015150Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5015428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.5015580Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.5015825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.5015913Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5015916Z 2025-11-03T16:38:43.5016013Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5016204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5016263Z res = mod(**inputs) 2025-11-03T16:38:43.5016507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5016601Z outputs = self.roberta( 2025-11-03T16:38:43.5016847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5016923Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5017165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5017237Z layer_outputs = layer_module( 2025-11-03T16:38:43.5017445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5017517Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5017753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5017816Z return func(*args, **kwargs) 2025-11-03T16:38:43.5018090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5018167Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5018395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5018464Z return func(*args, **kwargs) 2025-11-03T16:38:43.5018706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5018778Z self_outputs = self.self( 2025-11-03T16:38:43.5019004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5019067Z return func(*args, **kwargs) 2025-11-03T16:38:43.5019322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.5019524Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.5019527Z 2025-11-03T16:38:43.5019637Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5019877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5019943Z res = mod(**inputs) 2025-11-03T16:38:43.5020193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5020256Z outputs = self.roberta( 2025-11-03T16:38:43.5020516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5020583Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5020837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5020904Z layer_outputs = layer_module( 2025-11-03T16:38:43.5021117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5021228Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5021453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5021522Z return func(*args, **kwargs) 2025-11-03T16:38:43.5021768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5021854Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5022084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5022147Z return func(*args, **kwargs) 2025-11-03T16:38:43.5022417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5022484Z self_outputs = self.self( 2025-11-03T16:38:43.5022723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5022791Z return func(*args, **kwargs) 2025-11-03T16:38:43.5023040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.5023116Z self.key(current_states) 2025-11-03T16:38:43.5023119Z 2025-11-03T16:38:43.5023218Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5023411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5023473Z res = mod(**inputs) 2025-11-03T16:38:43.5023719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5023804Z outputs = self.roberta( 2025-11-03T16:38:43.5024051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5024131Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5024378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5024455Z layer_outputs = layer_module( 2025-11-03T16:38:43.5024664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5024739Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5024973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5025037Z return func(*args, **kwargs) 2025-11-03T16:38:43.5025291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5025369Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5025598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5025686Z return func(*args, **kwargs) 2025-11-03T16:38:43.5025932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5026005Z self_outputs = self.self( 2025-11-03T16:38:43.5026237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5026302Z return func(*args, **kwargs) 2025-11-03T16:38:43.5026560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.5026628Z self.value(current_states) 2025-11-03T16:38:43.5026633Z 2025-11-03T16:38:43.5026722Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.5026820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5027034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5027096Z res = mod(**inputs) 2025-11-03T16:38:43.5027346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5027417Z outputs = self.roberta( 2025-11-03T16:38:43.5027667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5027740Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5027988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5028054Z layer_outputs = layer_module( 2025-11-03T16:38:43.5028286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5028364Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5028607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5028672Z return func(*args, **kwargs) 2025-11-03T16:38:43.5028922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5029005Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5029237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5029308Z return func(*args, **kwargs) 2025-11-03T16:38:43.5029572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5029648Z self_outputs = self.self( 2025-11-03T16:38:43.5029877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5029943Z return func(*args, **kwargs) 2025-11-03T16:38:43.5030198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.5030325Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.5030328Z 2025-11-03T16:38:43.5030431Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5030619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5030680Z res = mod(**inputs) 2025-11-03T16:38:43.5030934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5030999Z outputs = self.roberta( 2025-11-03T16:38:43.5031254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5031347Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5031600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5031668Z layer_outputs = layer_module( 2025-11-03T16:38:43.5031877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5031960Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5032191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5032261Z return func(*args, **kwargs) 2025-11-03T16:38:43.5032507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5032587Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5032822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5032905Z return func(*args, **kwargs) 2025-11-03T16:38:43.5033164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.5033289Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.5033548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.5033627Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5033630Z 2025-11-03T16:38:43.5033730Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5033928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5034005Z res = mod(**inputs) 2025-11-03T16:38:43.5034334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5034408Z outputs = self.roberta( 2025-11-03T16:38:43.5034654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5034730Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5034982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5035059Z layer_outputs = layer_module( 2025-11-03T16:38:43.5035277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5035355Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5035617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5035686Z return func(*args, **kwargs) 2025-11-03T16:38:43.5035953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5036036Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5036308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5036382Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5036669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5036794Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5037045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.5037134Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5037137Z 2025-11-03T16:38:43.5037238Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5037470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5037541Z res = mod(**inputs) 2025-11-03T16:38:43.5037787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5037859Z outputs = self.roberta( 2025-11-03T16:38:43.5038110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5038183Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5038421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5038488Z layer_outputs = layer_module( 2025-11-03T16:38:43.5038704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5038794Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5039028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5039093Z return func(*args, **kwargs) 2025-11-03T16:38:43.5039335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5039420Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5039659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5039738Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5040032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5040154Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5040401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.5040506Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.5040713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.5040777Z return self.act(input) 2025-11-03T16:38:43.5040780Z 2025-11-03T16:38:43.5040878Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5041058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5041117Z res = mod(**inputs) 2025-11-03T16:38:43.5041380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5041444Z outputs = self.roberta( 2025-11-03T16:38:43.5041692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5041761Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5042003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5044354Z layer_outputs = layer_module( 2025-11-03T16:38:43.5044576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5044659Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5044896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5044962Z return func(*args, **kwargs) 2025-11-03T16:38:43.5045221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5045302Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5045551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5045625Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5045900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.5046054Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.5046308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.5046385Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5046389Z 2025-11-03T16:38:43.5046493Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5046679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5046763Z res = mod(**inputs) 2025-11-03T16:38:43.5047014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5047078Z outputs = self.roberta( 2025-11-03T16:38:43.5047325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5047393Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5047633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5047709Z layer_outputs = layer_module( 2025-11-03T16:38:43.5047913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5048009Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5048233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5048299Z return func(*args, **kwargs) 2025-11-03T16:38:43.5048548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5048624Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5048855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5048917Z return func(*args, **kwargs) 2025-11-03T16:38:43.5049164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5049231Z self_outputs = self.self( 2025-11-03T16:38:43.5049468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5049539Z return func(*args, **kwargs) 2025-11-03T16:38:43.5049779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.5049982Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.5049986Z 2025-11-03T16:38:43.5050103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5050343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5050407Z res = mod(**inputs) 2025-11-03T16:38:43.5050649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5050716Z outputs = self.roberta( 2025-11-03T16:38:43.5050962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5051035Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5051276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5051342Z layer_outputs = layer_module( 2025-11-03T16:38:43.5051555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5051630Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5051860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5051923Z return func(*args, **kwargs) 2025-11-03T16:38:43.5052162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5052246Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5052471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5052557Z return func(*args, **kwargs) 2025-11-03T16:38:43.5052798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5052864Z self_outputs = self.self( 2025-11-03T16:38:43.5053096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5053159Z return func(*args, **kwargs) 2025-11-03T16:38:43.5053406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.5053471Z self.key(current_states) 2025-11-03T16:38:43.5053474Z 2025-11-03T16:38:43.5053577Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5053776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5053837Z res = mod(**inputs) 2025-11-03T16:38:43.5054087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5054150Z outputs = self.roberta( 2025-11-03T16:38:43.5054397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5054466Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5054705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5054780Z layer_outputs = layer_module( 2025-11-03T16:38:43.5054986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5055092Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5055322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5055387Z return func(*args, **kwargs) 2025-11-03T16:38:43.5055638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5055713Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5055946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5056047Z return func(*args, **kwargs) 2025-11-03T16:38:43.5056303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5056370Z self_outputs = self.self( 2025-11-03T16:38:43.5056600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5056675Z return func(*args, **kwargs) 2025-11-03T16:38:43.5056924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.5056999Z self.value(current_states) 2025-11-03T16:38:43.5057002Z 2025-11-03T16:38:43.5057079Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.5057175Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5057372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5057433Z res = mod(**inputs) 2025-11-03T16:38:43.5057687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5057750Z outputs = self.roberta( 2025-11-03T16:38:43.5057999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5058073Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5058322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5058415Z layer_outputs = layer_module( 2025-11-03T16:38:43.5058626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5058707Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5058937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5058999Z return func(*args, **kwargs) 2025-11-03T16:38:43.5059259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5059335Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5059591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5059654Z return func(*args, **kwargs) 2025-11-03T16:38:43.5059899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5059971Z self_outputs = self.self( 2025-11-03T16:38:43.5060194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5060266Z return func(*args, **kwargs) 2025-11-03T16:38:43.5060507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.5060640Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.5060643Z 2025-11-03T16:38:43.5060738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5060937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5061006Z res = mod(**inputs) 2025-11-03T16:38:43.5061253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5061325Z outputs = self.roberta( 2025-11-03T16:38:43.5061571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5061660Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5061921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5061987Z layer_outputs = layer_module( 2025-11-03T16:38:43.5062201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5062275Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5062500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5062573Z return func(*args, **kwargs) 2025-11-03T16:38:43.5062812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5062894Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5063118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5063188Z return func(*args, **kwargs) 2025-11-03T16:38:43.5063429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.5063551Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.5063808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.5063886Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5063905Z 2025-11-03T16:38:43.5064009Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5064197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5064258Z res = mod(**inputs) 2025-11-03T16:38:43.5064515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5064593Z outputs = self.roberta( 2025-11-03T16:38:43.5064838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5064903Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5065218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5065300Z layer_outputs = layer_module( 2025-11-03T16:38:43.5065512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5065596Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5065824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5065895Z return func(*args, **kwargs) 2025-11-03T16:38:43.5066143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5066222Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5066470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5066544Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5066845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5066961Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5067219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.5067297Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5067300Z 2025-11-03T16:38:43.5067399Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5067611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5067673Z res = mod(**inputs) 2025-11-03T16:38:43.5067926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5067991Z outputs = self.roberta( 2025-11-03T16:38:43.5068239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5068314Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5068562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5068637Z layer_outputs = layer_module( 2025-11-03T16:38:43.5068847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5068923Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5069161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5069227Z return func(*args, **kwargs) 2025-11-03T16:38:43.5069482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5069563Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5069816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5069908Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5070186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5070310Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5070560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.5070674Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.5070878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.5070944Z return self.act(input) 2025-11-03T16:38:43.5070956Z 2025-11-03T16:38:43.5071056Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5071258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5071328Z res = mod(**inputs) 2025-11-03T16:38:43.5071576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5071646Z outputs = self.roberta( 2025-11-03T16:38:43.5071892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5071961Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5072215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5072282Z layer_outputs = layer_module( 2025-11-03T16:38:43.5072511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5072589Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5072823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5072897Z return func(*args, **kwargs) 2025-11-03T16:38:43.5073146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5073232Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5073487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5073559Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5073844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.5073977Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.5074320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.5074407Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5074411Z 2025-11-03T16:38:43.5074519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5074713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5074777Z res = mod(**inputs) 2025-11-03T16:38:43.5075041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5075108Z outputs = self.roberta( 2025-11-03T16:38:43.5075371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5075441Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5075698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5075775Z layer_outputs = layer_module( 2025-11-03T16:38:43.5076011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5076097Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5076339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5076415Z return func(*args, **kwargs) 2025-11-03T16:38:43.5076671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5076751Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5076997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5077065Z return func(*args, **kwargs) 2025-11-03T16:38:43.5077354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5077424Z self_outputs = self.self( 2025-11-03T16:38:43.5077665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5077740Z return func(*args, **kwargs) 2025-11-03T16:38:43.5077998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.5078212Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.5078216Z 2025-11-03T16:38:43.5078315Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5078515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5078594Z res = mod(**inputs) 2025-11-03T16:38:43.5078853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5078930Z outputs = self.roberta( 2025-11-03T16:38:43.5079185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5079261Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5079515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5079604Z layer_outputs = layer_module( 2025-11-03T16:38:43.5079831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5079906Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5080155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5080221Z return func(*args, **kwargs) 2025-11-03T16:38:43.5080478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5080566Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5080803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5080879Z return func(*args, **kwargs) 2025-11-03T16:38:43.5081134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5081208Z self_outputs = self.self( 2025-11-03T16:38:43.5081446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5081513Z return func(*args, **kwargs) 2025-11-03T16:38:43.5081778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.5081864Z self.key(current_states) 2025-11-03T16:38:43.5081868Z 2025-11-03T16:38:43.5081976Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5082167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5082229Z res = mod(**inputs) 2025-11-03T16:38:43.5082492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5082558Z outputs = self.roberta( 2025-11-03T16:38:43.5082820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5082892Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5083169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5083239Z layer_outputs = layer_module( 2025-11-03T16:38:43.5083455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5083538Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5083785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5083861Z return func(*args, **kwargs) 2025-11-03T16:38:43.5084112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5084191Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5084434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5084514Z return func(*args, **kwargs) 2025-11-03T16:38:43.5084780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5084850Z self_outputs = self.self( 2025-11-03T16:38:43.5085088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5085161Z return func(*args, **kwargs) 2025-11-03T16:38:43.5085421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.5085517Z self.value(current_states) 2025-11-03T16:38:43.5085521Z 2025-11-03T16:38:43.5085608Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.5085709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5085899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5085970Z res = mod(**inputs) 2025-11-03T16:38:43.5086228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5086303Z outputs = self.roberta( 2025-11-03T16:38:43.5086559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5086629Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5086903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5086973Z layer_outputs = layer_module( 2025-11-03T16:38:43.5087196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5087272Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5087517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5087584Z return func(*args, **kwargs) 2025-11-03T16:38:43.5087840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5087943Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5088179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5088251Z return func(*args, **kwargs) 2025-11-03T16:38:43.5088505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5088574Z self_outputs = self.self( 2025-11-03T16:38:43.5088818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5088885Z return func(*args, **kwargs) 2025-11-03T16:38:43.5089165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.5089300Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.5089305Z 2025-11-03T16:38:43.5089413Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5089606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5089670Z res = mod(**inputs) 2025-11-03T16:38:43.5089932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5090011Z outputs = self.roberta( 2025-11-03T16:38:43.5090273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5090342Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5090609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5090688Z layer_outputs = layer_module( 2025-11-03T16:38:43.5090906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5090991Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5091227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5091294Z return func(*args, **kwargs) 2025-11-03T16:38:43.5091580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5091660Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5091908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5091973Z return func(*args, **kwargs) 2025-11-03T16:38:43.5092239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.5092366Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.5092625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.5092715Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5092719Z 2025-11-03T16:38:43.5092821Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5093020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5093083Z res = mod(**inputs) 2025-11-03T16:38:43.5093342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5093413Z outputs = self.roberta( 2025-11-03T16:38:43.5093676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5093750Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5094005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5094077Z layer_outputs = layer_module( 2025-11-03T16:38:43.5094284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5094357Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5094590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5094655Z return func(*args, **kwargs) 2025-11-03T16:38:43.5094903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5094997Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5095237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5095316Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5095583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5095704Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5095948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.5096029Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5096039Z 2025-11-03T16:38:43.5096135Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5096320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5096404Z res = mod(**inputs) 2025-11-03T16:38:43.5096647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5096718Z outputs = self.roberta( 2025-11-03T16:38:43.5096961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5097029Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5097280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5097363Z layer_outputs = layer_module( 2025-11-03T16:38:43.5097576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5097649Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5097875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5097947Z return func(*args, **kwargs) 2025-11-03T16:38:43.5098187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5098276Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5098510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5098590Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5098874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5098987Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5099234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.5099343Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.5099547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.5099638Z return self.act(input) 2025-11-03T16:38:43.5099642Z 2025-11-03T16:38:43.5099736Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5099923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5099985Z res = mod(**inputs) 2025-11-03T16:38:43.5100232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5100294Z outputs = self.roberta( 2025-11-03T16:38:43.5100542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5100608Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5100866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5100942Z layer_outputs = layer_module( 2025-11-03T16:38:43.5101148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5101227Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5101452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5101517Z return func(*args, **kwargs) 2025-11-03T16:38:43.5101763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5101839Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5102102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5102176Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5102446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.5102580Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.5102823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.5102925Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5102928Z 2025-11-03T16:38:43.5103025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5103213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5103272Z res = mod(**inputs) 2025-11-03T16:38:43.5103517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5103588Z outputs = self.roberta( 2025-11-03T16:38:43.5103827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5103902Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5104143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5104208Z layer_outputs = layer_module( 2025-11-03T16:38:43.5104423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5104497Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5104729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5104792Z return func(*args, **kwargs) 2025-11-03T16:38:43.5105041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5105116Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5105358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5105431Z return func(*args, **kwargs) 2025-11-03T16:38:43.5105672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5105743Z self_outputs = self.self( 2025-11-03T16:38:43.5105966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5106029Z return func(*args, **kwargs) 2025-11-03T16:38:43.5106274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-11-03T16:38:43.5106482Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-11-03T16:38:43.5106486Z 2025-11-03T16:38:43.5106591Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5106776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5106843Z res = mod(**inputs) 2025-11-03T16:38:43.5107088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5107156Z outputs = self.roberta( 2025-11-03T16:38:43.5107404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5107471Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5107720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5107801Z layer_outputs = layer_module( 2025-11-03T16:38:43.5108011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5108094Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5108318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5108390Z return func(*args, **kwargs) 2025-11-03T16:38:43.5108632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5108722Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5108951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5109014Z return func(*args, **kwargs) 2025-11-03T16:38:43.5109263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5109330Z self_outputs = self.self( 2025-11-03T16:38:43.5109569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5109635Z return func(*args, **kwargs) 2025-11-03T16:38:43.5109881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward 2025-11-03T16:38:43.5109955Z self.key(current_states) 2025-11-03T16:38:43.5109960Z 2025-11-03T16:38:43.5110058Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5110252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5110315Z res = mod(**inputs) 2025-11-03T16:38:43.5110562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5110637Z outputs = self.roberta( 2025-11-03T16:38:43.5110884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5110978Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5111224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5111292Z layer_outputs = layer_module( 2025-11-03T16:38:43.5111511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5111589Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5111826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5111890Z return func(*args, **kwargs) 2025-11-03T16:38:43.5112145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5112237Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5112468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5112541Z return func(*args, **kwargs) 2025-11-03T16:38:43.5112787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5112860Z self_outputs = self.self( 2025-11-03T16:38:43.5113088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5113152Z return func(*args, **kwargs) 2025-11-03T16:38:43.5113539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 341, in forward 2025-11-03T16:38:43.5113610Z self.value(current_states) 2025-11-03T16:38:43.5113614Z 2025-11-03T16:38:43.5113747Z cudagraph partition due to non gpu ops 2025-11-03T16:38:43.5113850Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5114050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5114156Z res = mod(**inputs) 2025-11-03T16:38:43.5114418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5114496Z outputs = self.roberta( 2025-11-03T16:38:43.5114796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5114874Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5115133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5115200Z layer_outputs = layer_module( 2025-11-03T16:38:43.5115439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5115516Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5115757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5115822Z return func(*args, **kwargs) 2025-11-03T16:38:43.5116073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5116162Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5116393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5116465Z return func(*args, **kwargs) 2025-11-03T16:38:43.5116714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 439, in forward 2025-11-03T16:38:43.5116790Z self_outputs = self.self( 2025-11-03T16:38:43.5117022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5117111Z return func(*args, **kwargs) 2025-11-03T16:38:43.5117371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 362, in forward 2025-11-03T16:38:43.5117500Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-11-03T16:38:43.5117505Z 2025-11-03T16:38:43.5117611Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5117797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5117858Z res = mod(**inputs) 2025-11-03T16:38:43.5118114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5118181Z outputs = self.roberta( 2025-11-03T16:38:43.5118455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5118525Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5118775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5118847Z layer_outputs = layer_module( 2025-11-03T16:38:43.5119060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5119142Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5119374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5119444Z return func(*args, **kwargs) 2025-11-03T16:38:43.5119713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 512, in forward 2025-11-03T16:38:43.5119792Z self_attention_outputs = self.attention( 2025-11-03T16:38:43.5120030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5120096Z return func(*args, **kwargs) 2025-11-03T16:38:43.5120352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 448, in forward 2025-11-03T16:38:43.5120474Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:38:43.5120748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 386, in forward 2025-11-03T16:38:43.5120835Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5120838Z 2025-11-03T16:38:43.5120933Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5121129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5121190Z res = mod(**inputs) 2025-11-03T16:38:43.5121442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5121508Z outputs = self.roberta( 2025-11-03T16:38:43.5121756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5121831Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5122081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5122154Z layer_outputs = layer_module( 2025-11-03T16:38:43.5122368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5122442Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5122687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5122753Z return func(*args, **kwargs) 2025-11-03T16:38:43.5123022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5123103Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5123345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5123428Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5123706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5123829Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5124082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 464, in forward 2025-11-03T16:38:43.5124183Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5124187Z 2025-11-03T16:38:43.5124284Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5124472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5124542Z res = mod(**inputs) 2025-11-03T16:38:43.5124790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5124862Z outputs = self.roberta( 2025-11-03T16:38:43.5125111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5125180Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5125478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5125558Z layer_outputs = layer_module( 2025-11-03T16:38:43.5125773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5125846Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5126083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5126146Z return func(*args, **kwargs) 2025-11-03T16:38:43.5126388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5126494Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5126736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5126814Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5127095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 550, in feed_forward_chunk 2025-11-03T16:38:43.5127206Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:38:43.5127455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-11-03T16:38:43.5127559Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:38:43.5127765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:38:43.5127831Z return self.act(input) 2025-11-03T16:38:43.5127835Z 2025-11-03T16:38:43.5127937Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5128119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5128180Z res = mod(**inputs) 2025-11-03T16:38:43.5128431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 970, in forward 2025-11-03T16:38:43.5128496Z outputs = self.roberta( 2025-11-03T16:38:43.5128742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 861, in forward 2025-11-03T16:38:43.5128827Z encoder_outputs = self.encoder( 2025-11-03T16:38:43.5129066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 605, in forward 2025-11-03T16:38:43.5129143Z layer_outputs = layer_module( 2025-11-03T16:38:43.5129348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:38:43.5129428Z return super().__call__(*args, **kwargs) 2025-11-03T16:38:43.5129651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:38:43.5129715Z return func(*args, **kwargs) 2025-11-03T16:38:43.5129980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 542, in forward 2025-11-03T16:38:43.5130058Z layer_output = apply_chunking_to_forward( 2025-11-03T16:38:43.5130305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:38:43.5130375Z return forward_fn(*input_tensors) 2025-11-03T16:38:43.5130654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-11-03T16:38:43.5130780Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:38:43.5131021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 478, in forward 2025-11-03T16:38:43.5131104Z hidden_states = self.dense(hidden_states) 2025-11-03T16:38:43.5131107Z 2025-11-03T16:38:43.5131242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5131434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5131497Z res = mod(**inputs) 2025-11-03T16:38:43.5131740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 987, in forward 2025-11-03T16:38:43.5131840Z prediction_scores = self.lm_head(sequence_output) 2025-11-03T16:38:43.5132094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1120, in forward 2025-11-03T16:38:43.5132185Z x = self.dense(features) 2025-11-03T16:38:43.5132188Z 2025-11-03T16:38:43.5132281Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5132469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5132528Z res = mod(**inputs) 2025-11-03T16:38:43.5132776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 987, in forward 2025-11-03T16:38:43.5132874Z prediction_scores = self.lm_head(sequence_output) 2025-11-03T16:38:43.5133126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1125, in forward 2025-11-03T16:38:43.5133194Z x = self.decoder(x) 2025-11-03T16:38:43.5133198Z 2025-11-03T16:38:43.5133289Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:38:43.5133476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:38:43.5133542Z res = mod(**inputs) 2025-11-03T16:38:43.5133787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 993, in forward 2025-11-03T16:38:43.5133859Z lm_loss = self.loss_function( 2025-11-03T16:38:43.5134084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-11-03T16:38:43.5134254Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-11-03T16:38:43.5134503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-11-03T16:38:43.5134683Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-11-03T16:38:43.5134687Z 2025-11-03T16:38:53.1447260Z Compilation time (from dynamo_timed): 15.885332317 2025-11-03T16:38:53.1569301Z pass 2025-11-03T16:38:53.1571315Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:38:53.1572202Z TIMING: _recursive_pre_grad_passes:0.00639 _recursive_joint_graph_passes:0.62606 _recursive_post_grad_passes:0.06164 async_compile.wait:0.66782 code_gen:8.50662 inductor_compile:9.75456 backend_compile:12.6965 gc:0.00094 entire_frame_compile:15.88533 total_wall_time:15.88533 2025-11-03T16:38:53.1576764Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:7129 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:1966 2025-11-03T16:38:53.1581207Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-11-03T16:38:55.4349119Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:38:55.4350581Z import pynvml # type: ignore[import] 2025-11-03T16:38:58.6820387Z 2025-11-03T16:38:59.4975406Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:38:59.4979812Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:38:59.4984957Z cpu eval T5ForConditionalGeneration 2025-11-03T16:39:00.5876933Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:01.0331282Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:01.4298010Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:09.5282865Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5283438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5283832Z res = mod(**inputs) 2025-11-03T16:39:09.5284601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5285163Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5285532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5285894Z layer_outputs = layer_module( 2025-11-03T16:39:09.5286252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5286602Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5286975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5287330Z return func(*args, **kwargs) 2025-11-03T16:39:09.5287671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5288041Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5288533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5289051Z return func(*args, **kwargs) 2025-11-03T16:39:09.5289397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5289761Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5290147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5290600Z return func(*args, **kwargs) 2025-11-03T16:39:09.5290941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 548, in forward 2025-11-03T16:39:09.5291327Z position_bias = position_bias + causal_mask 2025-11-03T16:39:09.5291476Z 2025-11-03T16:39:09.5291589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5291946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5292258Z res = mod(**inputs) 2025-11-03T16:39:09.5292592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5292964Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5293562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5294114Z layer_outputs = layer_module( 2025-11-03T16:39:09.5294556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5295041Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5295594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5296030Z return func(*args, **kwargs) 2025-11-03T16:39:09.5296463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5296836Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5297197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5297559Z return func(*args, **kwargs) 2025-11-03T16:39:09.5297967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5298336Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5298707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5299061Z return func(*args, **kwargs) 2025-11-03T16:39:09.5299412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5299823Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5299956Z 2025-11-03T16:39:09.5300070Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5300513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5300819Z res = mod(**inputs) 2025-11-03T16:39:09.5301166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5301525Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5301889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5302361Z layer_outputs = layer_module( 2025-11-03T16:39:09.5302698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5303055Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5303423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5303779Z return func(*args, **kwargs) 2025-11-03T16:39:09.5304117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5304478Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5304850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5305234Z return func(*args, **kwargs) 2025-11-03T16:39:09.5305568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5305920Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5306286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5306641Z return func(*args, **kwargs) 2025-11-03T16:39:09.5306998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5307365Z key_states = self.k(current_states) 2025-11-03T16:39:09.5307503Z 2025-11-03T16:39:09.5307608Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5307986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5308313Z res = mod(**inputs) 2025-11-03T16:39:09.5308728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5309088Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5309450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5309816Z layer_outputs = layer_module( 2025-11-03T16:39:09.5310174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5310523Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5310904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5311355Z return func(*args, **kwargs) 2025-11-03T16:39:09.5311824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5312236Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5312605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5312979Z return func(*args, **kwargs) 2025-11-03T16:39:09.5313516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5313951Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5314451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5314822Z return func(*args, **kwargs) 2025-11-03T16:39:09.5315183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5315568Z value_states = self.v(current_states) 2025-11-03T16:39:09.5315701Z 2025-11-03T16:39:09.5315791Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5316010Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5316334Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5316849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5317191Z res = mod(**inputs) 2025-11-03T16:39:09.5317545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5317898Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5318246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5318602Z layer_outputs = layer_module( 2025-11-03T16:39:09.5318938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5319283Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5319646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5320044Z return func(*args, **kwargs) 2025-11-03T16:39:09.5320392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5320757Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5321119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5321558Z return func(*args, **kwargs) 2025-11-03T16:39:09.5321988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5322353Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5322767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5323120Z return func(*args, **kwargs) 2025-11-03T16:39:09.5323456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5323812Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5323933Z 2025-11-03T16:39:09.5324046Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5324526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5324833Z res = mod(**inputs) 2025-11-03T16:39:09.5325161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5325511Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5325879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5326223Z layer_outputs = layer_module( 2025-11-03T16:39:09.5326550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5327000Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5327405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5327744Z return func(*args, **kwargs) 2025-11-03T16:39:09.5328126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5328491Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5328858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5329245Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5329718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5330249Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5330443Z 2025-11-03T16:39:09.5330544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5330885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5331184Z res = mod(**inputs) 2025-11-03T16:39:09.5331498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5331845Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5332188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5332532Z layer_outputs = layer_module( 2025-11-03T16:39:09.5332851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5333201Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5333570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5334045Z return func(*args, **kwargs) 2025-11-03T16:39:09.5334450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5334806Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5335167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5335549Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5335927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5336277Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5336404Z 2025-11-03T16:39:09.5336522Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5337018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5337341Z res = mod(**inputs) 2025-11-03T16:39:09.5337664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5338007Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5338351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5338698Z layer_outputs = layer_module( 2025-11-03T16:39:09.5339027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5339369Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5339742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5340150Z return func(*args, **kwargs) 2025-11-03T16:39:09.5340580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5340943Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5341297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5341704Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5342081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5342439Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5342566Z 2025-11-03T16:39:09.5342672Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5343061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5343491Z res = mod(**inputs) 2025-11-03T16:39:09.5343812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5344163Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5344555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5344910Z layer_outputs = layer_module( 2025-11-03T16:39:09.5345246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5345600Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5345984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5346424Z return func(*args, **kwargs) 2025-11-03T16:39:09.5346767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5347130Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5347526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5347884Z return func(*args, **kwargs) 2025-11-03T16:39:09.5348219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5348672Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5349057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5349436Z return func(*args, **kwargs) 2025-11-03T16:39:09.5349781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5350143Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5350278Z 2025-11-03T16:39:09.5350394Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5350739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5351046Z res = mod(**inputs) 2025-11-03T16:39:09.5351375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5351746Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5352110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5352518Z layer_outputs = layer_module( 2025-11-03T16:39:09.5352899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5353260Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5353661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5354186Z return func(*args, **kwargs) 2025-11-03T16:39:09.5354585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5354974Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5355357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5355768Z return func(*args, **kwargs) 2025-11-03T16:39:09.5356134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5356500Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5356864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5357257Z return func(*args, **kwargs) 2025-11-03T16:39:09.5357653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5358010Z key_states = self.k(current_states) 2025-11-03T16:39:09.5358135Z 2025-11-03T16:39:09.5358242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5358577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5358885Z res = mod(**inputs) 2025-11-03T16:39:09.5359211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5359569Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5359911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5360264Z layer_outputs = layer_module( 2025-11-03T16:39:09.5360599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5360951Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5361311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5361685Z return func(*args, **kwargs) 2025-11-03T16:39:09.5362031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5362394Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5362764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5363114Z return func(*args, **kwargs) 2025-11-03T16:39:09.5363445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5363804Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5364185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5364536Z return func(*args, **kwargs) 2025-11-03T16:39:09.5364868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5365225Z value_states = self.v(current_states) 2025-11-03T16:39:09.5365359Z 2025-11-03T16:39:09.5365436Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5365644Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5365868Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5366201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5366509Z res = mod(**inputs) 2025-11-03T16:39:09.5366835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5367211Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5367554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5367908Z layer_outputs = layer_module( 2025-11-03T16:39:09.5368237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5368582Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5368944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5369311Z return func(*args, **kwargs) 2025-11-03T16:39:09.5369650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5370010Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5370375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5370732Z return func(*args, **kwargs) 2025-11-03T16:39:09.5371065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5371428Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5371792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5372145Z return func(*args, **kwargs) 2025-11-03T16:39:09.5372475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5372847Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5372976Z 2025-11-03T16:39:09.5373075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5373415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5373723Z res = mod(**inputs) 2025-11-03T16:39:09.5374043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5374415Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5374772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5375132Z layer_outputs = layer_module( 2025-11-03T16:39:09.5375464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5375824Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5376190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5376555Z return func(*args, **kwargs) 2025-11-03T16:39:09.5376901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5377294Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5377673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5378078Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5378488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5378855Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5378986Z 2025-11-03T16:39:09.5379086Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5379434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5379747Z res = mod(**inputs) 2025-11-03T16:39:09.5380083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5380479Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5381179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5381543Z layer_outputs = layer_module( 2025-11-03T16:39:09.5381884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5382238Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5382601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5382990Z return func(*args, **kwargs) 2025-11-03T16:39:09.5383343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5383735Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5384129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5384529Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5384929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5385299Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5385432Z 2025-11-03T16:39:09.5385543Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5385893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5386205Z res = mod(**inputs) 2025-11-03T16:39:09.5386541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5386907Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5387264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5387620Z layer_outputs = layer_module( 2025-11-03T16:39:09.5387961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5388346Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5388718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5389086Z return func(*args, **kwargs) 2025-11-03T16:39:09.5389432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5389813Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5390189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5390596Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5391012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5391382Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5391522Z 2025-11-03T16:39:09.5391625Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5391977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5392292Z res = mod(**inputs) 2025-11-03T16:39:09.5392623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5392990Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5393352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5393707Z layer_outputs = layer_module( 2025-11-03T16:39:09.5394038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5394499Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5394890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5395263Z return func(*args, **kwargs) 2025-11-03T16:39:09.5395625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5395985Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5396379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5396735Z return func(*args, **kwargs) 2025-11-03T16:39:09.5397079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5397446Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5397811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5398165Z return func(*args, **kwargs) 2025-11-03T16:39:09.5398507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5398866Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5398993Z 2025-11-03T16:39:09.5399099Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5399435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5399741Z res = mod(**inputs) 2025-11-03T16:39:09.5400070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5400425Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5400767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5401126Z layer_outputs = layer_module( 2025-11-03T16:39:09.5401460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5401832Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5402208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5402566Z return func(*args, **kwargs) 2025-11-03T16:39:09.5402915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5403284Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5403655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5404008Z return func(*args, **kwargs) 2025-11-03T16:39:09.5404377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5404738Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5405104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5405458Z return func(*args, **kwargs) 2025-11-03T16:39:09.5405791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5406152Z key_states = self.k(current_states) 2025-11-03T16:39:09.5406285Z 2025-11-03T16:39:09.5406381Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5406715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5407013Z res = mod(**inputs) 2025-11-03T16:39:09.5407326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5407689Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5408034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5408379Z layer_outputs = layer_module( 2025-11-03T16:39:09.5408695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5409032Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5409385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5409753Z return func(*args, **kwargs) 2025-11-03T16:39:09.5410084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5410432Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5410792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5411134Z return func(*args, **kwargs) 2025-11-03T16:39:09.5411466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5411823Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5412169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5412512Z return func(*args, **kwargs) 2025-11-03T16:39:09.5412844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5413193Z value_states = self.v(current_states) 2025-11-03T16:39:09.5413444Z 2025-11-03T16:39:09.5413524Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5413728Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5413951Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5414297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5414593Z res = mod(**inputs) 2025-11-03T16:39:09.5414968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5415313Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5415654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5415998Z layer_outputs = layer_module( 2025-11-03T16:39:09.5416314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5416654Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5417005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5417352Z return func(*args, **kwargs) 2025-11-03T16:39:09.5417709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5418056Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5418409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5418752Z return func(*args, **kwargs) 2025-11-03T16:39:09.5419081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5419425Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5419780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5420126Z return func(*args, **kwargs) 2025-11-03T16:39:09.5420479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5420831Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5420954Z 2025-11-03T16:39:09.5421049Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5421383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5421686Z res = mod(**inputs) 2025-11-03T16:39:09.5422006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5422377Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5422709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5423053Z layer_outputs = layer_module( 2025-11-03T16:39:09.5423383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5423720Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5424065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5424410Z return func(*args, **kwargs) 2025-11-03T16:39:09.5424739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5425098Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5425459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5425838Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5426217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5426564Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5426687Z 2025-11-03T16:39:09.5426790Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5427119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5427410Z res = mod(**inputs) 2025-11-03T16:39:09.5427750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5428097Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5428444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5428783Z layer_outputs = layer_module( 2025-11-03T16:39:09.5429109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5429447Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5429799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5430148Z return func(*args, **kwargs) 2025-11-03T16:39:09.5430503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5430874Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5431236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5431624Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5432000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5432356Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5432492Z 2025-11-03T16:39:09.5432589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5432929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5433235Z res = mod(**inputs) 2025-11-03T16:39:09.5433575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5433935Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5434368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5434737Z layer_outputs = layer_module( 2025-11-03T16:39:09.5435083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5435460Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5435836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5436190Z return func(*args, **kwargs) 2025-11-03T16:39:09.5436529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5436895Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5437271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5437667Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5438059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5438421Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5438551Z 2025-11-03T16:39:09.5438649Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5438993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5439307Z res = mod(**inputs) 2025-11-03T16:39:09.5439634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5439992Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5440341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5440719Z layer_outputs = layer_module( 2025-11-03T16:39:09.5441054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5441405Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5441763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5442123Z return func(*args, **kwargs) 2025-11-03T16:39:09.5442467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5442826Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5443191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5443558Z return func(*args, **kwargs) 2025-11-03T16:39:09.5443899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5444263Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5444628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5444974Z return func(*args, **kwargs) 2025-11-03T16:39:09.5445316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5445672Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5445798Z 2025-11-03T16:39:09.5445904Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5446247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5446542Z res = mod(**inputs) 2025-11-03T16:39:09.5446892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5447253Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5447603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5447961Z layer_outputs = layer_module( 2025-11-03T16:39:09.5448288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5448655Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5449022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5449383Z return func(*args, **kwargs) 2025-11-03T16:39:09.5449729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5450090Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5450450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5450799Z return func(*args, **kwargs) 2025-11-03T16:39:09.5451131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5451482Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5451845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5452192Z return func(*args, **kwargs) 2025-11-03T16:39:09.5452524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5452874Z key_states = self.k(current_states) 2025-11-03T16:39:09.5452998Z 2025-11-03T16:39:09.5453095Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5453431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5453753Z res = mod(**inputs) 2025-11-03T16:39:09.5454071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5454410Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5454755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5455099Z layer_outputs = layer_module( 2025-11-03T16:39:09.5455422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5455757Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5456107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5456450Z return func(*args, **kwargs) 2025-11-03T16:39:09.5456797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5457151Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5457498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5457851Z return func(*args, **kwargs) 2025-11-03T16:39:09.5458184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5458539Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5458893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5459231Z return func(*args, **kwargs) 2025-11-03T16:39:09.5459589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5459944Z value_states = self.v(current_states) 2025-11-03T16:39:09.5460069Z 2025-11-03T16:39:09.5460152Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5460351Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5460564Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5460897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5461193Z res = mod(**inputs) 2025-11-03T16:39:09.5461531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5461869Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5462210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5462552Z layer_outputs = layer_module( 2025-11-03T16:39:09.5462878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5463218Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5463568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5463911Z return func(*args, **kwargs) 2025-11-03T16:39:09.5464244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5464596Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5464941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5465287Z return func(*args, **kwargs) 2025-11-03T16:39:09.5465618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5465971Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5466328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5466689Z return func(*args, **kwargs) 2025-11-03T16:39:09.5467019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5467367Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5467490Z 2025-11-03T16:39:09.5467594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5467927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5468229Z res = mod(**inputs) 2025-11-03T16:39:09.5468551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5468900Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5469265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5469607Z layer_outputs = layer_module( 2025-11-03T16:39:09.5469935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5470274Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5470629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5470975Z return func(*args, **kwargs) 2025-11-03T16:39:09.5471301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5471653Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5472009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5472370Z return func(*args, **kwargs) 2025-11-03T16:39:09.5472713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 612, in forward 2025-11-03T16:39:09.5473114Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:39:09.5473296Z 2025-11-03T16:39:09.5473391Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5473719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5474011Z res = mod(**inputs) 2025-11-03T16:39:09.5474439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5474807Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5475170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5475524Z layer_outputs = layer_module( 2025-11-03T16:39:09.5475861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5476206Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5476572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5476929Z return func(*args, **kwargs) 2025-11-03T16:39:09.5477271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5477640Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5478014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5478413Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5478808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5479169Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5479298Z 2025-11-03T16:39:09.5479396Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5479791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5480112Z res = mod(**inputs) 2025-11-03T16:39:09.5480451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5480818Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5481177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5481540Z layer_outputs = layer_module( 2025-11-03T16:39:09.5481882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5482244Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5482632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5482998Z return func(*args, **kwargs) 2025-11-03T16:39:09.5483345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5483722Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5484089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5484478Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5484869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5485227Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5485356Z 2025-11-03T16:39:09.5485461Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5485826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5486130Z res = mod(**inputs) 2025-11-03T16:39:09.5486466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5486826Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5487181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5487552Z layer_outputs = layer_module( 2025-11-03T16:39:09.5487883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5488228Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5488588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5488941Z return func(*args, **kwargs) 2025-11-03T16:39:09.5489277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5489648Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5490014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5490407Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5490791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5491144Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5491275Z 2025-11-03T16:39:09.5491370Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5491702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5492002Z res = mod(**inputs) 2025-11-03T16:39:09.5492316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5492661Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5493024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5493378Z layer_outputs = layer_module( 2025-11-03T16:39:09.5493704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5494043Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5494400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5494752Z return func(*args, **kwargs) 2025-11-03T16:39:09.5495087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5495437Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5495812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5496163Z return func(*args, **kwargs) 2025-11-03T16:39:09.5496493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5496846Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5497196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5497544Z return func(*args, **kwargs) 2025-11-03T16:39:09.5497874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5498237Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5498361Z 2025-11-03T16:39:09.5498463Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5498810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5499112Z res = mod(**inputs) 2025-11-03T16:39:09.5499434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5499778Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5500110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5500471Z layer_outputs = layer_module( 2025-11-03T16:39:09.5500792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5501129Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5501482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5501820Z return func(*args, **kwargs) 2025-11-03T16:39:09.5502152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5502506Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5502858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5503194Z return func(*args, **kwargs) 2025-11-03T16:39:09.5503522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5503878Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5504231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5504575Z return func(*args, **kwargs) 2025-11-03T16:39:09.5504896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5505243Z key_states = self.k(current_states) 2025-11-03T16:39:09.5505372Z 2025-11-03T16:39:09.5505467Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5505820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5518564Z res = mod(**inputs) 2025-11-03T16:39:09.5518985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5519380Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5519744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5520095Z layer_outputs = layer_module( 2025-11-03T16:39:09.5520433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5520786Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5521285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5521657Z return func(*args, **kwargs) 2025-11-03T16:39:09.5521993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5522354Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5522718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5523072Z return func(*args, **kwargs) 2025-11-03T16:39:09.5523406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5523773Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5524137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5524531Z return func(*args, **kwargs) 2025-11-03T16:39:09.5524870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5525215Z value_states = self.v(current_states) 2025-11-03T16:39:09.5525355Z 2025-11-03T16:39:09.5525434Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5525637Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5525861Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5526249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5526558Z res = mod(**inputs) 2025-11-03T16:39:09.5526887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5527235Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5527580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5527919Z layer_outputs = layer_module( 2025-11-03T16:39:09.5528248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5528598Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5528959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5529306Z return func(*args, **kwargs) 2025-11-03T16:39:09.5529635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5529987Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5530344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5530693Z return func(*args, **kwargs) 2025-11-03T16:39:09.5531020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5531374Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5531778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5532135Z return func(*args, **kwargs) 2025-11-03T16:39:09.5532470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5532817Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5532948Z 2025-11-03T16:39:09.5533049Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5533393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5533700Z res = mod(**inputs) 2025-11-03T16:39:09.5534023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5534402Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5534753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5535104Z layer_outputs = layer_module( 2025-11-03T16:39:09.5535430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5535765Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5536120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5536467Z return func(*args, **kwargs) 2025-11-03T16:39:09.5536798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5537163Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5537541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5537947Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5538337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5538696Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5538825Z 2025-11-03T16:39:09.5538926Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5539302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5539608Z res = mod(**inputs) 2025-11-03T16:39:09.5539935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5540283Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5540624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5540974Z layer_outputs = layer_module( 2025-11-03T16:39:09.5541299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5541644Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5541996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5542336Z return func(*args, **kwargs) 2025-11-03T16:39:09.5542669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5543034Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5543392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5543768Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5544155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5544531Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5544660Z 2025-11-03T16:39:09.5544766Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5545106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5545401Z res = mod(**inputs) 2025-11-03T16:39:09.5545728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5546079Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5546421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5546769Z layer_outputs = layer_module( 2025-11-03T16:39:09.5547105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5547460Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5547818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5548167Z return func(*args, **kwargs) 2025-11-03T16:39:09.5548492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5548855Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5549216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5549601Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5549985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5550346Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5550482Z 2025-11-03T16:39:09.5550581Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5550916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5551217Z res = mod(**inputs) 2025-11-03T16:39:09.5551528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5551879Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5552240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5552591Z layer_outputs = layer_module( 2025-11-03T16:39:09.5552915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5553249Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5553606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5553959Z return func(*args, **kwargs) 2025-11-03T16:39:09.5554393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5554763Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5555128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5555518Z return func(*args, **kwargs) 2025-11-03T16:39:09.5555855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5556218Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5556579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5556943Z return func(*args, **kwargs) 2025-11-03T16:39:09.5557292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5557678Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5557808Z 2025-11-03T16:39:09.5557918Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5558253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5558564Z res = mod(**inputs) 2025-11-03T16:39:09.5558897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5559257Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5559611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5559957Z layer_outputs = layer_module( 2025-11-03T16:39:09.5560311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5560671Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5561044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5561399Z return func(*args, **kwargs) 2025-11-03T16:39:09.5561744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5562116Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5562490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5562863Z return func(*args, **kwargs) 2025-11-03T16:39:09.5563204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5563578Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5563970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5564330Z return func(*args, **kwargs) 2025-11-03T16:39:09.5564667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5565028Z key_states = self.k(current_states) 2025-11-03T16:39:09.5565163Z 2025-11-03T16:39:09.5565262Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5565625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5565939Z res = mod(**inputs) 2025-11-03T16:39:09.5566271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5566636Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5567001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5567365Z layer_outputs = layer_module( 2025-11-03T16:39:09.5567704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5568063Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5568427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5568781Z return func(*args, **kwargs) 2025-11-03T16:39:09.5569123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5569474Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5569838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5570193Z return func(*args, **kwargs) 2025-11-03T16:39:09.5570536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5570897Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5571264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5571611Z return func(*args, **kwargs) 2025-11-03T16:39:09.5571941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5572295Z value_states = self.v(current_states) 2025-11-03T16:39:09.5572421Z 2025-11-03T16:39:09.5572503Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5572698Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5572920Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5573261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5573558Z res = mod(**inputs) 2025-11-03T16:39:09.5573899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5574248Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5574590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5574931Z layer_outputs = layer_module( 2025-11-03T16:39:09.5575253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5575594Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5575947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5576294Z return func(*args, **kwargs) 2025-11-03T16:39:09.5576634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5576990Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5577371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5577734Z return func(*args, **kwargs) 2025-11-03T16:39:09.5578073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5578432Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5578823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5579183Z return func(*args, **kwargs) 2025-11-03T16:39:09.5579524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5579879Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5580014Z 2025-11-03T16:39:09.5580118Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5580463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5580774Z res = mod(**inputs) 2025-11-03T16:39:09.5581113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5581479Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5581834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5582194Z layer_outputs = layer_module( 2025-11-03T16:39:09.5582525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5582874Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5583240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5583601Z return func(*args, **kwargs) 2025-11-03T16:39:09.5583955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5584352Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5584722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5585084Z return func(*args, **kwargs) 2025-11-03T16:39:09.5585436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5585812Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5586188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5586556Z return func(*args, **kwargs) 2025-11-03T16:39:09.5586922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5587293Z key_states = self.k(current_states) 2025-11-03T16:39:09.5587431Z 2025-11-03T16:39:09.5587534Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5587882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5588186Z res = mod(**inputs) 2025-11-03T16:39:09.5588521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5588879Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5589235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5589596Z layer_outputs = layer_module( 2025-11-03T16:39:09.5589926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5590310Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5590681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5591049Z return func(*args, **kwargs) 2025-11-03T16:39:09.5591397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5591764Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5592159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5592525Z return func(*args, **kwargs) 2025-11-03T16:39:09.5592870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5593236Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5593616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5593981Z return func(*args, **kwargs) 2025-11-03T16:39:09.5594414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5594794Z value_states = self.v(current_states) 2025-11-03T16:39:09.5594924Z 2025-11-03T16:39:09.5595003Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5595209Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5595449Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5595790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5596084Z res = mod(**inputs) 2025-11-03T16:39:09.5596409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5596760Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5597109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5597473Z layer_outputs = layer_module( 2025-11-03T16:39:09.5597786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5598124Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5598474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5598818Z return func(*args, **kwargs) 2025-11-03T16:39:09.5599140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5599491Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5599840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5600186Z return func(*args, **kwargs) 2025-11-03T16:39:09.5600529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5600878Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5601230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5601572Z return func(*args, **kwargs) 2025-11-03T16:39:09.5601899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5602244Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5602366Z 2025-11-03T16:39:09.5602461Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5602791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5603087Z res = mod(**inputs) 2025-11-03T16:39:09.5603424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5603763Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5604107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5604447Z layer_outputs = layer_module( 2025-11-03T16:39:09.5604768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5605119Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5605461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5605803Z return func(*args, **kwargs) 2025-11-03T16:39:09.5606146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5606500Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5606849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5607192Z return func(*args, **kwargs) 2025-11-03T16:39:09.5607518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5607878Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5608230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5608568Z return func(*args, **kwargs) 2025-11-03T16:39:09.5608897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5609244Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5609367Z 2025-11-03T16:39:09.5609472Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5609808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5610102Z res = mod(**inputs) 2025-11-03T16:39:09.5610439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5610786Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5611128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5611468Z layer_outputs = layer_module( 2025-11-03T16:39:09.5611791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5612131Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5612484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5612831Z return func(*args, **kwargs) 2025-11-03T16:39:09.5613173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5613671Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5614035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5614386Z return func(*args, **kwargs) 2025-11-03T16:39:09.5614719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5615074Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5615433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5615780Z return func(*args, **kwargs) 2025-11-03T16:39:09.5616164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5616509Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5616639Z 2025-11-03T16:39:09.5616736Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5617073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5617375Z res = mod(**inputs) 2025-11-03T16:39:09.5617693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5618057Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5618398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5618746Z layer_outputs = layer_module( 2025-11-03T16:39:09.5619067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5619401Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5619754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5620100Z return func(*args, **kwargs) 2025-11-03T16:39:09.5620429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5620776Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5621123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5621467Z return func(*args, **kwargs) 2025-11-03T16:39:09.5621799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 612, in forward 2025-11-03T16:39:09.5622198Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:39:09.5622370Z 2025-11-03T16:39:09.5622471Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5622801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5623102Z res = mod(**inputs) 2025-11-03T16:39:09.5623451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5623797Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5624131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5624478Z layer_outputs = layer_module( 2025-11-03T16:39:09.5624800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5625139Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5625491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5625833Z return func(*args, **kwargs) 2025-11-03T16:39:09.5626191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5626560Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5626919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5627307Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5627684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5628038Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5628171Z 2025-11-03T16:39:09.5628268Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5628610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5628906Z res = mod(**inputs) 2025-11-03T16:39:09.5629264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5629715Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5630185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5630690Z layer_outputs = layer_module( 2025-11-03T16:39:09.5631161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5631744Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5632311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5632873Z return func(*args, **kwargs) 2025-11-03T16:39:09.5633393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5634024Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5634746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5635423Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5635897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5636258Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5636401Z 2025-11-03T16:39:09.5636501Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5636849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5637219Z res = mod(**inputs) 2025-11-03T16:39:09.5637563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1726, in forward 2025-11-03T16:39:09.5637920Z encoder_outputs = self.encoder( 2025-11-03T16:39:09.5638280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5638680Z layer_outputs = layer_module( 2025-11-03T16:39:09.5639016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5639358Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5639725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5640082Z return func(*args, **kwargs) 2025-11-03T16:39:09.5640422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5640795Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5641159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5641592Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5641980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5642338Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5642469Z 2025-11-03T16:39:09.5642574Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5642907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5643217Z res = mod(**inputs) 2025-11-03T16:39:09.5643546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5643901Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5644245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5644619Z layer_outputs = layer_module( 2025-11-03T16:39:09.5644952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5645300Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5645658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5646009Z return func(*args, **kwargs) 2025-11-03T16:39:09.5646350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5646743Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5647110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5647467Z return func(*args, **kwargs) 2025-11-03T16:39:09.5647804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5648173Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5648544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5648911Z return func(*args, **kwargs) 2025-11-03T16:39:09.5649236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5649587Z key_states = self.k(current_states) 2025-11-03T16:39:09.5649716Z 2025-11-03T16:39:09.5649812Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5650150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5650451Z res = mod(**inputs) 2025-11-03T16:39:09.5650763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5651112Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5651455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5651822Z layer_outputs = layer_module( 2025-11-03T16:39:09.5652142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5652484Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5652842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5653192Z return func(*args, **kwargs) 2025-11-03T16:39:09.5653524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5653872Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5654234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5654598Z return func(*args, **kwargs) 2025-11-03T16:39:09.5654929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5655287Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5655636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5655984Z return func(*args, **kwargs) 2025-11-03T16:39:09.5656313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5656386Z value_states = self.v(current_states) 2025-11-03T16:39:09.5656390Z 2025-11-03T16:39:09.5656473Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5656546Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5656648Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5656852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5656915Z res = mod(**inputs) 2025-11-03T16:39:09.5657146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5657215Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5657442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5657525Z layer_outputs = layer_module( 2025-11-03T16:39:09.5657736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5657818Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5658043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5658115Z return func(*args, **kwargs) 2025-11-03T16:39:09.5658338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5658413Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5658647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5658710Z return func(*args, **kwargs) 2025-11-03T16:39:09.5658937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5659017Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5659248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5659311Z return func(*args, **kwargs) 2025-11-03T16:39:09.5659529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5659611Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5659615Z 2025-11-03T16:39:09.5659713Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5659919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5659979Z res = mod(**inputs) 2025-11-03T16:39:09.5660197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5660272Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5660492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5660565Z layer_outputs = layer_module( 2025-11-03T16:39:09.5660771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5660846Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5661099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5661163Z return func(*args, **kwargs) 2025-11-03T16:39:09.5661389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5661473Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5661698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5661809Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5662027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5662109Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5662112Z 2025-11-03T16:39:09.5662206Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5662416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5662477Z res = mod(**inputs) 2025-11-03T16:39:09.5662702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5662776Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5662999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5663088Z layer_outputs = layer_module( 2025-11-03T16:39:09.5663294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5663373Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5663605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5663669Z return func(*args, **kwargs) 2025-11-03T16:39:09.5663903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5663991Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5664214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5664322Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5664544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5664630Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5664633Z 2025-11-03T16:39:09.5664727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5664914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5664972Z res = mod(**inputs) 2025-11-03T16:39:09.5665195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5665271Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5665509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5665581Z layer_outputs = layer_module( 2025-11-03T16:39:09.5665786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5665867Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5666088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5666151Z return func(*args, **kwargs) 2025-11-03T16:39:09.5666374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5666459Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5666700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5666808Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5667026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5667106Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5667110Z 2025-11-03T16:39:09.5667204Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5667390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5667447Z res = mod(**inputs) 2025-11-03T16:39:09.5667673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5667738Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5667980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5668053Z layer_outputs = layer_module( 2025-11-03T16:39:09.5668265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5668344Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5668573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5668654Z return func(*args, **kwargs) 2025-11-03T16:39:09.5668890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5668968Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5669206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5669271Z return func(*args, **kwargs) 2025-11-03T16:39:09.5669501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5669589Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5669815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5669887Z return func(*args, **kwargs) 2025-11-03T16:39:09.5670108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5670189Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5670193Z 2025-11-03T16:39:09.5670289Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5670477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5670546Z res = mod(**inputs) 2025-11-03T16:39:09.5670775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5670850Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5671107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5671175Z layer_outputs = layer_module( 2025-11-03T16:39:09.5671395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5671472Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5671708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5671773Z return func(*args, **kwargs) 2025-11-03T16:39:09.5671999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5672084Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5672333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5672411Z return func(*args, **kwargs) 2025-11-03T16:39:09.5672640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5672729Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5672961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5673028Z return func(*args, **kwargs) 2025-11-03T16:39:09.5673260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5673334Z key_states = self.k(current_states) 2025-11-03T16:39:09.5673338Z 2025-11-03T16:39:09.5673442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5673650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5673714Z res = mod(**inputs) 2025-11-03T16:39:09.5673953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5674022Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5674336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5674434Z layer_outputs = layer_module( 2025-11-03T16:39:09.5674647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5674735Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5674977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5675056Z return func(*args, **kwargs) 2025-11-03T16:39:09.5675296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5675389Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5675691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5675777Z return func(*args, **kwargs) 2025-11-03T16:39:09.5676172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5676302Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5676730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5676833Z return func(*args, **kwargs) 2025-11-03T16:39:09.5677242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5677358Z value_states = self.v(current_states) 2025-11-03T16:39:09.5677363Z 2025-11-03T16:39:09.5677482Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5677637Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5677800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5678137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5678239Z res = mod(**inputs) 2025-11-03T16:39:09.5678656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5678780Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5679187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5679307Z layer_outputs = layer_module( 2025-11-03T16:39:09.5679691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5679841Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5680269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5680367Z return func(*args, **kwargs) 2025-11-03T16:39:09.5680614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5680693Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5680938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5681014Z return func(*args, **kwargs) 2025-11-03T16:39:09.5681250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5681349Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5681614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5681680Z return func(*args, **kwargs) 2025-11-03T16:39:09.5681938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5682014Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5682018Z 2025-11-03T16:39:09.5682127Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5682375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5682447Z res = mod(**inputs) 2025-11-03T16:39:09.5682695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5682765Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5683007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5683076Z layer_outputs = layer_module( 2025-11-03T16:39:09.5683302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5683382Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5683627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5683699Z return func(*args, **kwargs) 2025-11-03T16:39:09.5683947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5684030Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5684261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5684325Z return func(*args, **kwargs) 2025-11-03T16:39:09.5684562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5684642Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5684899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5684962Z return func(*args, **kwargs) 2025-11-03T16:39:09.5685191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5685268Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5685271Z 2025-11-03T16:39:09.5685366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5685560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5685621Z res = mod(**inputs) 2025-11-03T16:39:09.5685854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5685942Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5686172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5686249Z layer_outputs = layer_module( 2025-11-03T16:39:09.5686466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5686546Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5686776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5686840Z return func(*args, **kwargs) 2025-11-03T16:39:09.5687073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5687151Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5687404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5687470Z return func(*args, **kwargs) 2025-11-03T16:39:09.5687707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5687787Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5688016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5688105Z return func(*args, **kwargs) 2025-11-03T16:39:09.5688330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5688412Z key_states = self.k(current_states) 2025-11-03T16:39:09.5688416Z 2025-11-03T16:39:09.5688512Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5688701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5688771Z res = mod(**inputs) 2025-11-03T16:39:09.5689006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5689082Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5689307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5689375Z layer_outputs = layer_module( 2025-11-03T16:39:09.5689599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5689670Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5689903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5689965Z return func(*args, **kwargs) 2025-11-03T16:39:09.5690190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5690264Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5690503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5690572Z return func(*args, **kwargs) 2025-11-03T16:39:09.5690789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5690874Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5691095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5691158Z return func(*args, **kwargs) 2025-11-03T16:39:09.5691384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5691456Z value_states = self.v(current_states) 2025-11-03T16:39:09.5691460Z 2025-11-03T16:39:09.5691555Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5691629Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5691725Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5691913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5691971Z res = mod(**inputs) 2025-11-03T16:39:09.5692201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5692269Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5692493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5692558Z layer_outputs = layer_module( 2025-11-03T16:39:09.5692763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5692859Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5693084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5693156Z return func(*args, **kwargs) 2025-11-03T16:39:09.5693373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5693448Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5693720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5693783Z return func(*args, **kwargs) 2025-11-03T16:39:09.5694009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5694087Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5694309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5694380Z return func(*args, **kwargs) 2025-11-03T16:39:09.5694599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5694678Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5694682Z 2025-11-03T16:39:09.5694774Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5694965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5695025Z res = mod(**inputs) 2025-11-03T16:39:09.5695246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5695321Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5695538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5695611Z layer_outputs = layer_module( 2025-11-03T16:39:09.5695815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5695902Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5696128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5696190Z return func(*args, **kwargs) 2025-11-03T16:39:09.5696413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5696500Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5696714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5696827Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5697060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5697141Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5697146Z 2025-11-03T16:39:09.5697239Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5697425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5697484Z res = mod(**inputs) 2025-11-03T16:39:09.5697703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5697777Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5697997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5698070Z layer_outputs = layer_module( 2025-11-03T16:39:09.5698274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5698362Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5698596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5698662Z return func(*args, **kwargs) 2025-11-03T16:39:09.5698893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5698977Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5699222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5699329Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5699551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5699634Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5699637Z 2025-11-03T16:39:09.5699735Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5699925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5699986Z res = mod(**inputs) 2025-11-03T16:39:09.5700210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5700286Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5700517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5700594Z layer_outputs = layer_module( 2025-11-03T16:39:09.5700800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5700873Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5701105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5701171Z return func(*args, **kwargs) 2025-11-03T16:39:09.5701397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5702270Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5702496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5702602Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5702823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5702907Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5702910Z 2025-11-03T16:39:09.5703004Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5703193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5703254Z res = mod(**inputs) 2025-11-03T16:39:09.5703496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5703575Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5703793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5703871Z layer_outputs = layer_module( 2025-11-03T16:39:09.5704074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5704153Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5704376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5704439Z return func(*args, **kwargs) 2025-11-03T16:39:09.5704678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5704757Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5704986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5705049Z return func(*args, **kwargs) 2025-11-03T16:39:09.5705273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5705358Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5705598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5705670Z return func(*args, **kwargs) 2025-11-03T16:39:09.5705890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5705965Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5705976Z 2025-11-03T16:39:09.5706077Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5706264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5706335Z res = mod(**inputs) 2025-11-03T16:39:09.5706559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5706637Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5706860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5706929Z layer_outputs = layer_module( 2025-11-03T16:39:09.5707143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5707221Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5707477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5707546Z return func(*args, **kwargs) 2025-11-03T16:39:09.5707773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5707878Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5708111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5708184Z return func(*args, **kwargs) 2025-11-03T16:39:09.5708413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5708495Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5708732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5708798Z return func(*args, **kwargs) 2025-11-03T16:39:09.5709048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5709122Z key_states = self.k(current_states) 2025-11-03T16:39:09.5709127Z 2025-11-03T16:39:09.5709230Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5709417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5709479Z res = mod(**inputs) 2025-11-03T16:39:09.5709716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5709785Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5710020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5710087Z layer_outputs = layer_module( 2025-11-03T16:39:09.5710316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5710399Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5710628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5710698Z return func(*args, **kwargs) 2025-11-03T16:39:09.5710922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5710999Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5711251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5711311Z return func(*args, **kwargs) 2025-11-03T16:39:09.5711537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5711614Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5711853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5711914Z return func(*args, **kwargs) 2025-11-03T16:39:09.5712139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5712217Z value_states = self.v(current_states) 2025-11-03T16:39:09.5712220Z 2025-11-03T16:39:09.5712296Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5712377Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5712475Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5712661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5712729Z res = mod(**inputs) 2025-11-03T16:39:09.5712960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5713037Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5713461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5713617Z layer_outputs = layer_module( 2025-11-03T16:39:09.5713848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5713928Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5714228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5714305Z return func(*args, **kwargs) 2025-11-03T16:39:09.5714548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5714629Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5714878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5714999Z return func(*args, **kwargs) 2025-11-03T16:39:09.5715226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5715313Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5715546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5715609Z return func(*args, **kwargs) 2025-11-03T16:39:09.5715842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5715918Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5715921Z 2025-11-03T16:39:09.5716028Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5716218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5716281Z res = mod(**inputs) 2025-11-03T16:39:09.5716556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5716629Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5716860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5716927Z layer_outputs = layer_module( 2025-11-03T16:39:09.5717143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5717245Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5717474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5717547Z return func(*args, **kwargs) 2025-11-03T16:39:09.5717770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5717852Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5718082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5718147Z return func(*args, **kwargs) 2025-11-03T16:39:09.5718377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 612, in forward 2025-11-03T16:39:09.5718506Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:39:09.5718511Z 2025-11-03T16:39:09.5718611Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5718795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5718859Z res = mod(**inputs) 2025-11-03T16:39:09.5719084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5719151Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5719387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5719473Z layer_outputs = layer_module( 2025-11-03T16:39:09.5719694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5719770Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5720002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5720077Z return func(*args, **kwargs) 2025-11-03T16:39:09.5720304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5720387Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5720615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5720694Z return func(*args, **kwargs) 2025-11-03T16:39:09.5720924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5721006Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5721242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5721305Z return func(*args, **kwargs) 2025-11-03T16:39:09.5721534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5721607Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5721610Z 2025-11-03T16:39:09.5721709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5721905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5721966Z res = mod(**inputs) 2025-11-03T16:39:09.5722217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5722289Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5722514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5722588Z layer_outputs = layer_module( 2025-11-03T16:39:09.5722796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5722895Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5723120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5723186Z return func(*args, **kwargs) 2025-11-03T16:39:09.5723415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5723492Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5723731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5723796Z return func(*args, **kwargs) 2025-11-03T16:39:09.5724033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5724113Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5724345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5724415Z return func(*args, **kwargs) 2025-11-03T16:39:09.5724630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5724705Z key_states = self.k(current_states) 2025-11-03T16:39:09.5724708Z 2025-11-03T16:39:09.5724803Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5724984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5725067Z res = mod(**inputs) 2025-11-03T16:39:09.5725286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5725359Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5725578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5725645Z layer_outputs = layer_module( 2025-11-03T16:39:09.5725855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5725927Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5726155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5726218Z return func(*args, **kwargs) 2025-11-03T16:39:09.5726457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5726534Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5726754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5726822Z return func(*args, **kwargs) 2025-11-03T16:39:09.5727040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5727125Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5727343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5727404Z return func(*args, **kwargs) 2025-11-03T16:39:09.5727639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5727715Z value_states = self.v(current_states) 2025-11-03T16:39:09.5727718Z 2025-11-03T16:39:09.5727802Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5727875Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5727969Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5728163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5728222Z res = mod(**inputs) 2025-11-03T16:39:09.5728466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5728533Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5728760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5728826Z layer_outputs = layer_module( 2025-11-03T16:39:09.5729036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5729116Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5729339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5729408Z return func(*args, **kwargs) 2025-11-03T16:39:09.5729625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5729700Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5729932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5729994Z return func(*args, **kwargs) 2025-11-03T16:39:09.5730216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5730295Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5730519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5730605Z return func(*args, **kwargs) 2025-11-03T16:39:09.5730823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5730901Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5730904Z 2025-11-03T16:39:09.5730998Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5731189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5731247Z res = mod(**inputs) 2025-11-03T16:39:09.5731469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5731544Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5731778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5731855Z layer_outputs = layer_module( 2025-11-03T16:39:09.5732060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5732131Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5732365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5732429Z return func(*args, **kwargs) 2025-11-03T16:39:09.5732653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5732737Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5732957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5733088Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5733311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5733388Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5733392Z 2025-11-03T16:39:09.5733487Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5733675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5733735Z res = mod(**inputs) 2025-11-03T16:39:09.5733974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5734048Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5734268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5734339Z layer_outputs = layer_module( 2025-11-03T16:39:09.5734547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5734619Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5734850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5734913Z return func(*args, **kwargs) 2025-11-03T16:39:09.5735136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5735220Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5735443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5735550Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5735768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5735852Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5735855Z 2025-11-03T16:39:09.5735945Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5736149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5736210Z res = mod(**inputs) 2025-11-03T16:39:09.5736432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5736508Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5736729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5736801Z layer_outputs = layer_module( 2025-11-03T16:39:09.5737006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5737076Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5737331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5737396Z return func(*args, **kwargs) 2025-11-03T16:39:09.5737622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5737705Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5737930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5738038Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5738254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5738334Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5738337Z 2025-11-03T16:39:09.5738433Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5738644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5738704Z res = mod(**inputs) 2025-11-03T16:39:09.5738924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5738998Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5739216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5739305Z layer_outputs = layer_module( 2025-11-03T16:39:09.5739511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5739585Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5739820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5739883Z return func(*args, **kwargs) 2025-11-03T16:39:09.5740117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5740194Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5740426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5740490Z return func(*args, **kwargs) 2025-11-03T16:39:09.5740708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5740795Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5741018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5741088Z return func(*args, **kwargs) 2025-11-03T16:39:09.5741306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5741379Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5741382Z 2025-11-03T16:39:09.5741485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5741689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5741757Z res = mod(**inputs) 2025-11-03T16:39:09.5741980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5742056Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5742277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5742344Z layer_outputs = layer_module( 2025-11-03T16:39:09.5742558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5742629Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5742871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5742934Z return func(*args, **kwargs) 2025-11-03T16:39:09.5743149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5743225Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5743447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5743518Z return func(*args, **kwargs) 2025-11-03T16:39:09.5743734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5743811Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5744044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5744122Z return func(*args, **kwargs) 2025-11-03T16:39:09.5744352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5744425Z key_states = self.k(current_states) 2025-11-03T16:39:09.5744428Z 2025-11-03T16:39:09.5744531Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5744714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5744791Z res = mod(**inputs) 2025-11-03T16:39:09.5745022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5745088Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5745333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5745400Z layer_outputs = layer_module( 2025-11-03T16:39:09.5745617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5745699Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5745935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5746005Z return func(*args, **kwargs) 2025-11-03T16:39:09.5746235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5746314Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5746554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5746619Z return func(*args, **kwargs) 2025-11-03T16:39:09.5746854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5746934Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5747175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5747257Z return func(*args, **kwargs) 2025-11-03T16:39:09.5747481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5747562Z value_states = self.v(current_states) 2025-11-03T16:39:09.5747566Z 2025-11-03T16:39:09.5747639Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5747720Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5747817Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5748006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5748075Z res = mod(**inputs) 2025-11-03T16:39:09.5748308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5748399Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5748653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5748719Z layer_outputs = layer_module( 2025-11-03T16:39:09.5748933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5749007Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5749241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5749303Z return func(*args, **kwargs) 2025-11-03T16:39:09.5749532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5749605Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5749855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5749926Z return func(*args, **kwargs) 2025-11-03T16:39:09.5750151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5750232Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5750461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5750540Z return func(*args, **kwargs) 2025-11-03T16:39:09.5750769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5750840Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5750844Z 2025-11-03T16:39:09.5750945Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5751132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5751191Z res = mod(**inputs) 2025-11-03T16:39:09.5751422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5751490Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5751723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5751790Z layer_outputs = layer_module( 2025-11-03T16:39:09.5752011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5752086Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5752318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5752389Z return func(*args, **kwargs) 2025-11-03T16:39:09.5752620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5752703Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5752950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5753014Z return func(*args, **kwargs) 2025-11-03T16:39:09.5753243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5753324Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5753561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5753625Z return func(*args, **kwargs) 2025-11-03T16:39:09.5753849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5753931Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5753936Z 2025-11-03T16:39:09.5754050Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5754314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5754384Z res = mod(**inputs) 2025-11-03T16:39:09.5754623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5754693Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5754918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5754998Z layer_outputs = layer_module( 2025-11-03T16:39:09.5755214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5755301Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5755562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5755642Z return func(*args, **kwargs) 2025-11-03T16:39:09.5755875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5755953Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5756193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5756275Z return func(*args, **kwargs) 2025-11-03T16:39:09.5756502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5756590Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5756867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5756936Z return func(*args, **kwargs) 2025-11-03T16:39:09.5757156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5757243Z key_states = self.k(current_states) 2025-11-03T16:39:09.5757246Z 2025-11-03T16:39:09.5757342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5757525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5757592Z res = mod(**inputs) 2025-11-03T16:39:09.5757814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5757889Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5758107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5758173Z layer_outputs = layer_module( 2025-11-03T16:39:09.5758386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5758460Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5758685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5758774Z return func(*args, **kwargs) 2025-11-03T16:39:09.5758994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5759075Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5759299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5759369Z return func(*args, **kwargs) 2025-11-03T16:39:09.5759593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5759677Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5759919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5759983Z return func(*args, **kwargs) 2025-11-03T16:39:09.5760211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5760286Z value_states = self.v(current_states) 2025-11-03T16:39:09.5760289Z 2025-11-03T16:39:09.5760371Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5760447Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5760545Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5760743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5760803Z res = mod(**inputs) 2025-11-03T16:39:09.5761037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5761122Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5761348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5761425Z layer_outputs = layer_module( 2025-11-03T16:39:09.5761636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5761717Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5762003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5762089Z return func(*args, **kwargs) 2025-11-03T16:39:09.5762313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5762388Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5762624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5762688Z return func(*args, **kwargs) 2025-11-03T16:39:09.5762919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5762997Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5763230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5763299Z return func(*args, **kwargs) 2025-11-03T16:39:09.5763528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5763607Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5763611Z 2025-11-03T16:39:09.5763706Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5763892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5763961Z res = mod(**inputs) 2025-11-03T16:39:09.5764186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5764280Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5764509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5764584Z layer_outputs = layer_module( 2025-11-03T16:39:09.5764798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5764874Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5765121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5765185Z return func(*args, **kwargs) 2025-11-03T16:39:09.5765420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5765512Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5765743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5765816Z return func(*args, **kwargs) 2025-11-03T16:39:09.5766040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 651, in forward 2025-11-03T16:39:09.5766172Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-11-03T16:39:09.5766177Z 2025-11-03T16:39:09.5766274Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5766468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5766528Z res = mod(**inputs) 2025-11-03T16:39:09.5766755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5766845Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5767072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5767149Z layer_outputs = layer_module( 2025-11-03T16:39:09.5767359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5767432Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5767669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5767750Z return func(*args, **kwargs) 2025-11-03T16:39:09.5767980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5768066Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5768296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5768415Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5768643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5768726Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5768729Z 2025-11-03T16:39:09.5768828Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5769023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5769085Z res = mod(**inputs) 2025-11-03T16:39:09.5769315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5769390Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5769617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5769692Z layer_outputs = layer_module( 2025-11-03T16:39:09.5769906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5769997Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5770232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5770295Z return func(*args, **kwargs) 2025-11-03T16:39:09.5770525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5770612Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5770833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5770950Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5771188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5771273Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5771278Z 2025-11-03T16:39:09.5771374Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5771565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5771626Z res = mod(**inputs) 2025-11-03T16:39:09.5771851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5771928Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5772150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5772224Z layer_outputs = layer_module( 2025-11-03T16:39:09.5772482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5772559Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5772795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5772862Z return func(*args, **kwargs) 2025-11-03T16:39:09.5773093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5773178Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5773420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5773531Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5773755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5773840Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5773843Z 2025-11-03T16:39:09.5773941Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5774131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5774194Z res = mod(**inputs) 2025-11-03T16:39:09.5774418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5774494Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5774719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5774799Z layer_outputs = layer_module( 2025-11-03T16:39:09.5775008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5775083Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5775321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5775388Z return func(*args, **kwargs) 2025-11-03T16:39:09.5775617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5775714Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5775950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5776016Z return func(*args, **kwargs) 2025-11-03T16:39:09.5776247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5776332Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5776563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5776633Z return func(*args, **kwargs) 2025-11-03T16:39:09.5776875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5776950Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5776955Z 2025-11-03T16:39:09.5777057Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5777244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5777312Z res = mod(**inputs) 2025-11-03T16:39:09.5777539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5777615Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5777843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5777911Z layer_outputs = layer_module( 2025-11-03T16:39:09.5778145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5778223Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5778460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5778526Z return func(*args, **kwargs) 2025-11-03T16:39:09.5778754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5778837Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5779089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5779161Z return func(*args, **kwargs) 2025-11-03T16:39:09.5779386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5779466Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5779705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5779771Z return func(*args, **kwargs) 2025-11-03T16:39:09.5780002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5780073Z key_states = self.k(current_states) 2025-11-03T16:39:09.5780077Z 2025-11-03T16:39:09.5780182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5780370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5780433Z res = mod(**inputs) 2025-11-03T16:39:09.5780668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5780735Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5780970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5781039Z layer_outputs = layer_module( 2025-11-03T16:39:09.5781251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5781353Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5781585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5781659Z return func(*args, **kwargs) 2025-11-03T16:39:09.5781892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5781971Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5782217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5782284Z return func(*args, **kwargs) 2025-11-03T16:39:09.5782541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5782623Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5782876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5782942Z return func(*args, **kwargs) 2025-11-03T16:39:09.5783182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5783268Z value_states = self.v(current_states) 2025-11-03T16:39:09.5783271Z 2025-11-03T16:39:09.5783349Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5783435Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5783534Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5783734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5783807Z res = mod(**inputs) 2025-11-03T16:39:09.5784063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5784144Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5784379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5784447Z layer_outputs = layer_module( 2025-11-03T16:39:09.5784674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5784810Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5785047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5785112Z return func(*args, **kwargs) 2025-11-03T16:39:09.5785338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5785423Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5785653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5785722Z return func(*args, **kwargs) 2025-11-03T16:39:09.5785951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5786031Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5786265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5786328Z return func(*args, **kwargs) 2025-11-03T16:39:09.5786564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5786635Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5786639Z 2025-11-03T16:39:09.5786745Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5786942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5787006Z res = mod(**inputs) 2025-11-03T16:39:09.5787267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5787340Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5787582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5787653Z layer_outputs = layer_module( 2025-11-03T16:39:09.5787877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5787955Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5788192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5788266Z return func(*args, **kwargs) 2025-11-03T16:39:09.5788516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5788605Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5788844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5788909Z return func(*args, **kwargs) 2025-11-03T16:39:09.5789145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5789231Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5789475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5789540Z return func(*args, **kwargs) 2025-11-03T16:39:09.5789786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5789873Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5789876Z 2025-11-03T16:39:09.5789975Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5790181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5790242Z res = mod(**inputs) 2025-11-03T16:39:09.5790482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5790567Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5790799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5790875Z layer_outputs = layer_module( 2025-11-03T16:39:09.5791093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5791179Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5791417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5791486Z return func(*args, **kwargs) 2025-11-03T16:39:09.5791723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5791800Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5792043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5792109Z return func(*args, **kwargs) 2025-11-03T16:39:09.5792341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5792430Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5792667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5792738Z return func(*args, **kwargs) 2025-11-03T16:39:09.5792970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5793066Z key_states = self.k(current_states) 2025-11-03T16:39:09.5793070Z 2025-11-03T16:39:09.5793165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5793356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5793425Z res = mod(**inputs) 2025-11-03T16:39:09.5793662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5793735Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5793972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5794043Z layer_outputs = layer_module( 2025-11-03T16:39:09.5794366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5794450Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5794697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5794763Z return func(*args, **kwargs) 2025-11-03T16:39:09.5794996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5795084Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5795321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5795396Z return func(*args, **kwargs) 2025-11-03T16:39:09.5795647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5795752Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5795982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5796049Z return func(*args, **kwargs) 2025-11-03T16:39:09.5796332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5796403Z value_states = self.v(current_states) 2025-11-03T16:39:09.5796424Z 2025-11-03T16:39:09.5796506Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5796577Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5796674Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5796864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5796923Z res = mod(**inputs) 2025-11-03T16:39:09.5797152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5797218Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5797438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5797513Z layer_outputs = layer_module( 2025-11-03T16:39:09.5797720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5797799Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5798020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5798089Z return func(*args, **kwargs) 2025-11-03T16:39:09.5798306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5798382Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5798618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5798696Z return func(*args, **kwargs) 2025-11-03T16:39:09.5798919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5798995Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5799220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5799293Z return func(*args, **kwargs) 2025-11-03T16:39:09.5799510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5799587Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5799590Z 2025-11-03T16:39:09.5799682Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5799880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5799947Z res = mod(**inputs) 2025-11-03T16:39:09.5800166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5800244Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5800471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5800542Z layer_outputs = layer_module( 2025-11-03T16:39:09.5800754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5800826Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5801057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5801119Z return func(*args, **kwargs) 2025-11-03T16:39:09.5801370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5801457Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5801676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5801792Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5802024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5802125Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5802129Z 2025-11-03T16:39:09.5802227Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5802424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5802486Z res = mod(**inputs) 2025-11-03T16:39:09.5802720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5802795Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5803024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5803098Z layer_outputs = layer_module( 2025-11-03T16:39:09.5803310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5803385Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5803624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5803689Z return func(*args, **kwargs) 2025-11-03T16:39:09.5803921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5804006Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5804236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5804370Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5804597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5804680Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5804684Z 2025-11-03T16:39:09.5804779Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5804976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5805037Z res = mod(**inputs) 2025-11-03T16:39:09.5805262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5805335Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5805581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5805656Z layer_outputs = layer_module( 2025-11-03T16:39:09.5805868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5805941Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5806179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5806246Z return func(*args, **kwargs) 2025-11-03T16:39:09.5806475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5806559Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5806782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5806916Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5807144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5807230Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5807233Z 2025-11-03T16:39:09.5807331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5807529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5807610Z res = mod(**inputs) 2025-11-03T16:39:09.5807838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5807915Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5808139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5808211Z layer_outputs = layer_module( 2025-11-03T16:39:09.5808425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5808498Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5808734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5808799Z return func(*args, **kwargs) 2025-11-03T16:39:09.5809041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5809125Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5809351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 344, in forward 2025-11-03T16:39:09.5809472Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-11-03T16:39:09.5809476Z 2025-11-03T16:39:09.5809570Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5809765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5809823Z res = mod(**inputs) 2025-11-03T16:39:09.5810068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5810134Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5810360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5810435Z layer_outputs = layer_module( 2025-11-03T16:39:09.5810641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5810720Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5810942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5811016Z return func(*args, **kwargs) 2025-11-03T16:39:09.5811259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5811337Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5811565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5811627Z return func(*args, **kwargs) 2025-11-03T16:39:09.5811851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5811931Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5812153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5812223Z return func(*args, **kwargs) 2025-11-03T16:39:09.5812440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5812535Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5812539Z 2025-11-03T16:39:09.5812634Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5812816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5812883Z res = mod(**inputs) 2025-11-03T16:39:09.5813102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5813191Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5813625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5813695Z layer_outputs = layer_module( 2025-11-03T16:39:09.5813908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5813981Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5814219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5814282Z return func(*args, **kwargs) 2025-11-03T16:39:09.5814509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5814584Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5814807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5814881Z return func(*args, **kwargs) 2025-11-03T16:39:09.5815098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5815183Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5815405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5815470Z return func(*args, **kwargs) 2025-11-03T16:39:09.5815698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5815816Z key_states = self.k(current_states) 2025-11-03T16:39:09.5815819Z 2025-11-03T16:39:09.5815922Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5816104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5816171Z res = mod(**inputs) 2025-11-03T16:39:09.5816397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5816464Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5816690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5816755Z layer_outputs = layer_module( 2025-11-03T16:39:09.5816992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5817067Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5817290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5817363Z return func(*args, **kwargs) 2025-11-03T16:39:09.5817581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5817665Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5817897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5817961Z return func(*args, **kwargs) 2025-11-03T16:39:09.5818189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5818289Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5818520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5818583Z return func(*args, **kwargs) 2025-11-03T16:39:09.5818807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5818879Z value_states = self.v(current_states) 2025-11-03T16:39:09.5818882Z 2025-11-03T16:39:09.5818954Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5819058Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5819151Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5819341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5819402Z res = mod(**inputs) 2025-11-03T16:39:09.5819626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5819700Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5819922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5819996Z layer_outputs = layer_module( 2025-11-03T16:39:09.5820203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5820277Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5820512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5820574Z return func(*args, **kwargs) 2025-11-03T16:39:09.5820798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 686, in forward 2025-11-03T16:39:09.5820873Z self_attention_outputs = self.layer[0]( 2025-11-03T16:39:09.5821099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5821170Z return func(*args, **kwargs) 2025-11-03T16:39:09.5821400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-11-03T16:39:09.5821483Z attention_output = self.SelfAttention( 2025-11-03T16:39:09.5821705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5821775Z return func(*args, **kwargs) 2025-11-03T16:39:09.5821992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5822063Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5822067Z 2025-11-03T16:39:09.5822170Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5822356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5822446Z res = mod(**inputs) 2025-11-03T16:39:09.5822669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5822737Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5822962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5823028Z layer_outputs = layer_module( 2025-11-03T16:39:09.5823245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5823320Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5823557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5823632Z return func(*args, **kwargs) 2025-11-03T16:39:09.5823880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5823967Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5824197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5824267Z return func(*args, **kwargs) 2025-11-03T16:39:09.5824495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5824592Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5824830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5824893Z return func(*args, **kwargs) 2025-11-03T16:39:09.5825132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-11-03T16:39:09.5825205Z query_states = self.q(hidden_states) 2025-11-03T16:39:09.5825210Z 2025-11-03T16:39:09.5825305Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5825497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5825558Z res = mod(**inputs) 2025-11-03T16:39:09.5825783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5825850Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5826074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5826141Z layer_outputs = layer_module( 2025-11-03T16:39:09.5826346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5826427Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5826651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5826720Z return func(*args, **kwargs) 2025-11-03T16:39:09.5826936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5827029Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5827267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5827333Z return func(*args, **kwargs) 2025-11-03T16:39:09.5827561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5827641Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5827866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5827941Z return func(*args, **kwargs) 2025-11-03T16:39:09.5828177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 512, in forward 2025-11-03T16:39:09.5828256Z key_states = self.k(current_states) 2025-11-03T16:39:09.5828261Z 2025-11-03T16:39:09.5828356Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5828549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5828608Z res = mod(**inputs) 2025-11-03T16:39:09.5828834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5828910Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5829133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5829204Z layer_outputs = layer_module( 2025-11-03T16:39:09.5829429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5829505Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5829734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5829799Z return func(*args, **kwargs) 2025-11-03T16:39:09.5830022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5830095Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5830339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5830412Z return func(*args, **kwargs) 2025-11-03T16:39:09.5830635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5830721Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5830956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5831027Z return func(*args, **kwargs) 2025-11-03T16:39:09.5831251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-11-03T16:39:09.5831326Z value_states = self.v(current_states) 2025-11-03T16:39:09.5831329Z 2025-11-03T16:39:09.5831412Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5831489Z cudagraph partition due to non gpu ops 2025-11-03T16:39:09.5831595Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5831782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5831844Z res = mod(**inputs) 2025-11-03T16:39:09.5832075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5832146Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5832377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5832463Z layer_outputs = layer_module( 2025-11-03T16:39:09.5832666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5832747Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5832968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5833039Z return func(*args, **kwargs) 2025-11-03T16:39:09.5833256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 710, in forward 2025-11-03T16:39:09.5833330Z cross_attention_outputs = self.layer[1]( 2025-11-03T16:39:09.5833560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5833638Z return func(*args, **kwargs) 2025-11-03T16:39:09.5833868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward 2025-11-03T16:39:09.5833949Z attention_output = self.EncDecAttention( 2025-11-03T16:39:09.5834256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5834327Z return func(*args, **kwargs) 2025-11-03T16:39:09.5834557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571, in forward 2025-11-03T16:39:09.5834637Z attn_output = self.o(attn_output) 2025-11-03T16:39:09.5834641Z 2025-11-03T16:39:09.5834738Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5834934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5835017Z res = mod(**inputs) 2025-11-03T16:39:09.5835291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5835368Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5835588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5835661Z layer_outputs = layer_module( 2025-11-03T16:39:09.5835866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5835966Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5836188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5836253Z return func(*args, **kwargs) 2025-11-03T16:39:09.5836483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5836574Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5836812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5836926Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5837144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-11-03T16:39:09.5837229Z hidden_states = self.wi(hidden_states) 2025-11-03T16:39:09.5837234Z 2025-11-03T16:39:09.5837329Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5837521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5837584Z res = mod(**inputs) 2025-11-03T16:39:09.5837806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5837886Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5838108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5838202Z layer_outputs = layer_module( 2025-11-03T16:39:09.5838407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5838488Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5838712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5838777Z return func(*args, **kwargs) 2025-11-03T16:39:09.5839001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5839085Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5839311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5839436Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5839655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-11-03T16:39:09.5839738Z hidden_states = self.act(hidden_states) 2025-11-03T16:39:09.5839741Z 2025-11-03T16:39:09.5839834Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5840020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5840081Z res = mod(**inputs) 2025-11-03T16:39:09.5840307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1763, in forward 2025-11-03T16:39:09.5840375Z decoder_outputs = self.decoder( 2025-11-03T16:39:09.5840593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1099, in forward 2025-11-03T16:39:09.5840682Z layer_outputs = layer_module( 2025-11-03T16:39:09.5840894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:09.5840975Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:09.5841201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:09.5841264Z return func(*args, **kwargs) 2025-11-03T16:39:09.5841492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward 2025-11-03T16:39:09.5841591Z hidden_states = self.layer[-1](hidden_states) 2025-11-03T16:39:09.5841818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-11-03T16:39:09.5841926Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-11-03T16:39:09.5842146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-11-03T16:39:09.5842227Z hidden_states = self.wo(hidden_states) 2025-11-03T16:39:09.5842232Z 2025-11-03T16:39:09.5842326Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5842515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5842573Z res = mod(**inputs) 2025-11-03T16:39:09.5842804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1792, in forward 2025-11-03T16:39:09.5842885Z lm_logits = self.lm_head(sequence_output) 2025-11-03T16:39:09.5842888Z 2025-11-03T16:39:09.5842981Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:09.5843172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:09.5843230Z res = mod(**inputs) 2025-11-03T16:39:09.5843463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1799, in forward 2025-11-03T16:39:09.5843594Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-11-03T16:39:09.5843621Z 2025-11-03T16:39:19.3803787Z Compilation time (from dynamo_timed): 17.184708544 2025-11-03T16:39:19.3985431Z pass 2025-11-03T16:39:19.3987638Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:19.3988573Z TIMING: _recursive_pre_grad_passes:0.01091 _recursive_joint_graph_passes:0.58089 _recursive_post_grad_passes:0.05897 async_compile.wait:0.70151 code_gen:9.13372 inductor_compile:10.37614 backend_compile:14.16308 gc:0.0001 entire_frame_compile:17.18471 total_wall_time:17.18471 2025-11-03T16:39:19.3989468Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:11388 | FakeTensor.__torch_dispatch__:4072 | ProxyTorchDispatchMode.__torch_dispatch__:3376 2025-11-03T16:39:19.3989952Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-11-03T16:39:21.6060183Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:39:21.6061118Z import pynvml # type: ignore[import] 2025-11-03T16:39:24.7084424Z 2025-11-03T16:39:25.5487454Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:39:25.5493177Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:39:25.5504908Z cpu eval T5Small 2025-11-03T16:39:26.7119810Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:27.0685181Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:27.4551416Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:38.9863053Z Compilation time (from dynamo_timed): 10.799306993 2025-11-03T16:39:39.0015213Z pass 2025-11-03T16:39:39.0020415Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:39.0025849Z TIMING: _recursive_pre_grad_passes:0.01033 async_compile.wait:0.00417 backend_compile:7.74717 gc:0.00014 entire_frame_compile:10.79931 total_wall_time:10.79931 2025-11-03T16:39:39.0027946Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:2289 | FakeTensor.__torch_dispatch__:17 2025-11-03T16:39:39.0033363Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-11-03T16:39:40.9261822Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:39:40.9262638Z import pynvml # type: ignore[import] 2025-11-03T16:39:44.0535423Z 2025-11-03T16:39:46.0987180Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:39:46.0988349Z loading model: 0it [00:02, ?it/s] 2025-11-03T16:39:46.1002726Z cpu eval TrOCRForCausalLM 2025-11-03T16:39:46.2615205Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-11-03T16:39:46.3002892Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:46.5547879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:46.8002889Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:39:53.8652836Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8657852Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8662476Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8662778Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8663023Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8663225Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8663763Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8663961Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8664159Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8664357Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8664552Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8664750Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8664983Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8665351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8665674Z res = mod(**inputs) 2025-11-03T16:39:53.8666060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8666536Z outputs = self.model.decoder( 2025-11-03T16:39:53.8666928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8667333Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8667699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8668051Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8668431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8668803Z return func(*args, **kwargs) 2025-11-03T16:39:53.8669175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8669581Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8670051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8670423Z return func(*args, **kwargs) 2025-11-03T16:39:53.8670803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8671279Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8671446Z 2025-11-03T16:39:53.8671556Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8671974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8672300Z res = mod(**inputs) 2025-11-03T16:39:53.8672670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8673077Z outputs = self.model.decoder( 2025-11-03T16:39:53.8673450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8673834Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8674295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8674672Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8675046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8675472Z return func(*args, **kwargs) 2025-11-03T16:39:53.8675859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8676281Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8676698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8677072Z return func(*args, **kwargs) 2025-11-03T16:39:53.8677458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8677861Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8678639Z 2025-11-03T16:39:53.8678756Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8679135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8679478Z res = mod(**inputs) 2025-11-03T16:39:53.8679866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8680296Z outputs = self.model.decoder( 2025-11-03T16:39:53.8680699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8681115Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8681494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8681917Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8682330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8682714Z return func(*args, **kwargs) 2025-11-03T16:39:53.8683102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8683532Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8683979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8684373Z return func(*args, **kwargs) 2025-11-03T16:39:53.8684744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8685146Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8685321Z 2025-11-03T16:39:53.8685408Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8685620Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8685828Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8686055Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8686411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8686732Z res = mod(**inputs) 2025-11-03T16:39:53.8687099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8687500Z outputs = self.model.decoder( 2025-11-03T16:39:53.8687876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8688391Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8688740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8689097Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8689469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8689835Z return func(*args, **kwargs) 2025-11-03T16:39:53.8690199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8690605Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8690998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8691359Z return func(*args, **kwargs) 2025-11-03T16:39:53.8691708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8692083Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8692216Z 2025-11-03T16:39:53.8692324Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8692672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8692982Z res = mod(**inputs) 2025-11-03T16:39:53.8693316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8693680Z outputs = self.model.decoder( 2025-11-03T16:39:53.8694044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8694412Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8694741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8695084Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8695467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8695826Z return func(*args, **kwargs) 2025-11-03T16:39:53.8696182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8696610Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8696788Z 2025-11-03T16:39:53.8696892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8697246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8697559Z res = mod(**inputs) 2025-11-03T16:39:53.8697907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8698288Z outputs = self.model.decoder( 2025-11-03T16:39:53.8698692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8699069Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8699399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8699753Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8700127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8700499Z return func(*args, **kwargs) 2025-11-03T16:39:53.8700882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8701306Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8701683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8702015Z return self.act(input) 2025-11-03T16:39:53.8702122Z 2025-11-03T16:39:53.8702235Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8702575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8702901Z res = mod(**inputs) 2025-11-03T16:39:53.8703266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8703652Z outputs = self.model.decoder( 2025-11-03T16:39:53.8704021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8704392Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8704733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8705092Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8705472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8705906Z return func(*args, **kwargs) 2025-11-03T16:39:53.8706263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8706687Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8706826Z 2025-11-03T16:39:53.8706926Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8707273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8707582Z res = mod(**inputs) 2025-11-03T16:39:53.8707938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8708322Z outputs = self.model.decoder( 2025-11-03T16:39:53.8708750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8709132Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8709516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8709876Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8710247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8710612Z return func(*args, **kwargs) 2025-11-03T16:39:53.8710974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8711376Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8711777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8712152Z return func(*args, **kwargs) 2025-11-03T16:39:53.8712559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8712990Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8713168Z 2025-11-03T16:39:53.8713617Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8713991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8714413Z res = mod(**inputs) 2025-11-03T16:39:53.8714854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8715327Z outputs = self.model.decoder( 2025-11-03T16:39:53.8715728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8716110Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8716459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8716819Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8717193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8717566Z return func(*args, **kwargs) 2025-11-03T16:39:53.8717937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8718349Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8718742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8719135Z return func(*args, **kwargs) 2025-11-03T16:39:53.8719500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8719896Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8720031Z 2025-11-03T16:39:53.8720146Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8720489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8720833Z res = mod(**inputs) 2025-11-03T16:39:53.8721184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8721561Z outputs = self.model.decoder( 2025-11-03T16:39:53.8721938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8722310Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8722654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8723016Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8723404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8723777Z return func(*args, **kwargs) 2025-11-03T16:39:53.8724135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8724533Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8724915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8725268Z return func(*args, **kwargs) 2025-11-03T16:39:53.8725621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8726003Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8726150Z 2025-11-03T16:39:53.8726227Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8726435Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8726635Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8726880Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8727223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8727535Z res = mod(**inputs) 2025-11-03T16:39:53.8727881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8728241Z outputs = self.model.decoder( 2025-11-03T16:39:53.8728608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8729009Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8729342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8729690Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8730046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8730406Z return func(*args, **kwargs) 2025-11-03T16:39:53.8730767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8731164Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8731539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8731893Z return func(*args, **kwargs) 2025-11-03T16:39:53.8732250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8732632Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8732762Z 2025-11-03T16:39:53.8732869Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8733206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8733517Z res = mod(**inputs) 2025-11-03T16:39:53.8733862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8734253Z outputs = self.model.decoder( 2025-11-03T16:39:53.8734653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8735029Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8735367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8735718Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8736082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8736438Z return func(*args, **kwargs) 2025-11-03T16:39:53.8736819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8737245Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8737412Z 2025-11-03T16:39:53.8737519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8737859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8738158Z res = mod(**inputs) 2025-11-03T16:39:53.8738502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8738877Z outputs = self.model.decoder( 2025-11-03T16:39:53.8739242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8739608Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8739959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8740311Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8740673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8741042Z return func(*args, **kwargs) 2025-11-03T16:39:53.8741389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8741791Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8742171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8742493Z return self.act(input) 2025-11-03T16:39:53.8742596Z 2025-11-03T16:39:53.8742700Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8743025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8743321Z res = mod(**inputs) 2025-11-03T16:39:53.8743655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8744016Z outputs = self.model.decoder( 2025-11-03T16:39:53.8744372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8744773Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8745098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8745437Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8745791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8746131Z return func(*args, **kwargs) 2025-11-03T16:39:53.8746480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8746851Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8746983Z 2025-11-03T16:39:53.8747110Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8747459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8747768Z res = mod(**inputs) 2025-11-03T16:39:53.8748115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8748497Z outputs = self.model.decoder( 2025-11-03T16:39:53.8748866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8749238Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8749579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8749934Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8750323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8750687Z return func(*args, **kwargs) 2025-11-03T16:39:53.8751039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8751438Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8751821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8752183Z return func(*args, **kwargs) 2025-11-03T16:39:53.8752534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8752948Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8753114Z 2025-11-03T16:39:53.8753231Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8753577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8753885Z res = mod(**inputs) 2025-11-03T16:39:53.8754283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8754662Z outputs = self.model.decoder( 2025-11-03T16:39:53.8755065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8755491Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8755844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8756202Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8756594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8756966Z return func(*args, **kwargs) 2025-11-03T16:39:53.8757337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8757733Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8758118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8758480Z return func(*args, **kwargs) 2025-11-03T16:39:53.8758844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8759262Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8759395Z 2025-11-03T16:39:53.8759499Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8759844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8760156Z res = mod(**inputs) 2025-11-03T16:39:53.8760505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8760892Z outputs = self.model.decoder( 2025-11-03T16:39:53.8761254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8761624Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8761960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8762309Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8762667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8763025Z return func(*args, **kwargs) 2025-11-03T16:39:53.8763384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8763795Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8764181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8764534Z return func(*args, **kwargs) 2025-11-03T16:39:53.8764893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8765279Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8765416Z 2025-11-03T16:39:53.8765502Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8765708Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8765905Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8766136Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8766480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8766810Z res = mod(**inputs) 2025-11-03T16:39:53.8767156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8767533Z outputs = self.model.decoder( 2025-11-03T16:39:53.8767904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8768278Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8768608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8768979Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8769349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8769712Z return func(*args, **kwargs) 2025-11-03T16:39:53.8770084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8770492Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8770892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8771268Z return func(*args, **kwargs) 2025-11-03T16:39:53.8771639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8772036Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8772175Z 2025-11-03T16:39:53.8772277Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8772629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8772947Z res = mod(**inputs) 2025-11-03T16:39:53.8773304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8773684Z outputs = self.model.decoder( 2025-11-03T16:39:53.8774069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8774468Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8774811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8775169Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8775537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8775909Z return func(*args, **kwargs) 2025-11-03T16:39:53.8776273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8776698Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8776865Z 2025-11-03T16:39:53.8776977Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8777335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8777651Z res = mod(**inputs) 2025-11-03T16:39:53.8778004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8778383Z outputs = self.model.decoder( 2025-11-03T16:39:53.8778748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8779127Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8779474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8779843Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8780243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8780613Z return func(*args, **kwargs) 2025-11-03T16:39:53.8780990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8781426Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8781824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8782172Z return self.act(input) 2025-11-03T16:39:53.8782304Z 2025-11-03T16:39:53.8782416Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8782796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8783151Z res = mod(**inputs) 2025-11-03T16:39:53.8783535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8783921Z outputs = self.model.decoder( 2025-11-03T16:39:53.8784310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8784697Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8785050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8785414Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8785789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8786168Z return func(*args, **kwargs) 2025-11-03T16:39:53.8786543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8786943Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8787083Z 2025-11-03T16:39:53.8787196Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8787561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8787873Z res = mod(**inputs) 2025-11-03T16:39:53.8788228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8788591Z outputs = self.model.decoder( 2025-11-03T16:39:53.8788950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8789324Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8789662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8790008Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8790373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8790726Z return func(*args, **kwargs) 2025-11-03T16:39:53.8791102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8791506Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8791896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8792246Z return func(*args, **kwargs) 2025-11-03T16:39:53.8792610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8793022Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8793181Z 2025-11-03T16:39:53.8793289Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8793637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8793950Z res = mod(**inputs) 2025-11-03T16:39:53.8794409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8794801Z outputs = self.model.decoder( 2025-11-03T16:39:53.8795180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8795559Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8795881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8796241Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8796600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8796952Z return func(*args, **kwargs) 2025-11-03T16:39:53.8797297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8797690Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8798066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8798421Z return func(*args, **kwargs) 2025-11-03T16:39:53.8798774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8799138Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8799277Z 2025-11-03T16:39:53.8799374Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8799712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8800016Z res = mod(**inputs) 2025-11-03T16:39:53.8800352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8800709Z outputs = self.model.decoder( 2025-11-03T16:39:53.8801066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8801444Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8801773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8802108Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8802462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8802809Z return func(*args, **kwargs) 2025-11-03T16:39:53.8803156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8803541Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8803903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8804262Z return func(*args, **kwargs) 2025-11-03T16:39:53.8804617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8804995Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8805129Z 2025-11-03T16:39:53.8805212Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8805412Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8805613Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8805836Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8806174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8806469Z res = mod(**inputs) 2025-11-03T16:39:53.8806811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8807201Z outputs = self.model.decoder( 2025-11-03T16:39:53.8807566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8807925Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8808245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8808586Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8808944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8809307Z return func(*args, **kwargs) 2025-11-03T16:39:53.8809645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8810022Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8810393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8810742Z return func(*args, **kwargs) 2025-11-03T16:39:53.8811087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8811450Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8811587Z 2025-11-03T16:39:53.8811686Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8812019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8812319Z res = mod(**inputs) 2025-11-03T16:39:53.8812647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8813007Z outputs = self.model.decoder( 2025-11-03T16:39:53.8813532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8813910Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8814245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8814632Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8814996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8815352Z return func(*args, **kwargs) 2025-11-03T16:39:53.8815714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8816122Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8816281Z 2025-11-03T16:39:53.8816380Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8816718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8817027Z res = mod(**inputs) 2025-11-03T16:39:53.8817396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8817756Z outputs = self.model.decoder( 2025-11-03T16:39:53.8818115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8818480Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8818811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8819158Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8819510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8819860Z return func(*args, **kwargs) 2025-11-03T16:39:53.8820210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8820644Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8821014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8821337Z return self.act(input) 2025-11-03T16:39:53.8821452Z 2025-11-03T16:39:53.8821556Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8821903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8822233Z res = mod(**inputs) 2025-11-03T16:39:53.8822562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8822924Z outputs = self.model.decoder( 2025-11-03T16:39:53.8823277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8823636Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8823968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8824303Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8824659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8825004Z return func(*args, **kwargs) 2025-11-03T16:39:53.8825353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8825727Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8825856Z 2025-11-03T16:39:53.8825953Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8826287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8826592Z res = mod(**inputs) 2025-11-03T16:39:53.8826929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8827283Z outputs = self.model.decoder( 2025-11-03T16:39:53.8827656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8828013Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8828337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8828680Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8829027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8829375Z return func(*args, **kwargs) 2025-11-03T16:39:53.8829727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8830113Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8830492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8830842Z return func(*args, **kwargs) 2025-11-03T16:39:53.8831191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8831593Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8831748Z 2025-11-03T16:39:53.8831855Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8832187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8832490Z res = mod(**inputs) 2025-11-03T16:39:53.8832824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8833186Z outputs = self.model.decoder( 2025-11-03T16:39:53.8833558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8833939Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8834351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8834716Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8835132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8835502Z return func(*args, **kwargs) 2025-11-03T16:39:53.8835857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8836249Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8836629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8836986Z return func(*args, **kwargs) 2025-11-03T16:39:53.8837338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8837719Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8837856Z 2025-11-03T16:39:53.8837954Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8838300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8838603Z res = mod(**inputs) 2025-11-03T16:39:53.8838937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8839305Z outputs = self.model.decoder( 2025-11-03T16:39:53.8839673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8840038Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8840362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8840718Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8841072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8841418Z return func(*args, **kwargs) 2025-11-03T16:39:53.8841763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8842143Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8842517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8842864Z return func(*args, **kwargs) 2025-11-03T16:39:53.8843210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8843611Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8843745Z 2025-11-03T16:39:53.8843820Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8844024Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8844225Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8844446Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8844773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8845077Z res = mod(**inputs) 2025-11-03T16:39:53.8845414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8845774Z outputs = self.model.decoder( 2025-11-03T16:39:53.8846123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8846499Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8846847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8847194Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8847553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8847897Z return func(*args, **kwargs) 2025-11-03T16:39:53.8848246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8848647Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8849019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8849363Z return func(*args, **kwargs) 2025-11-03T16:39:53.8849705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8850075Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8850208Z 2025-11-03T16:39:53.8850307Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8850639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8850930Z res = mod(**inputs) 2025-11-03T16:39:53.8851263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8851625Z outputs = self.model.decoder( 2025-11-03T16:39:53.8851976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8852335Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8852651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8852993Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8853352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8853719Z return func(*args, **kwargs) 2025-11-03T16:39:53.8854067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8854460Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8854626Z 2025-11-03T16:39:53.8854726Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8855060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8855358Z res = mod(**inputs) 2025-11-03T16:39:53.8855689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8856051Z outputs = self.model.decoder( 2025-11-03T16:39:53.8856432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8856799Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8857130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8857465Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8857824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8858173Z return func(*args, **kwargs) 2025-11-03T16:39:53.8858525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8858932Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8859294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8859635Z return self.act(input) 2025-11-03T16:39:53.8859750Z 2025-11-03T16:39:53.8859848Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8860189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8860484Z res = mod(**inputs) 2025-11-03T16:39:53.8860823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8861207Z outputs = self.model.decoder( 2025-11-03T16:39:53.8861564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8861922Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8862246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8862583Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8862942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8863292Z return func(*args, **kwargs) 2025-11-03T16:39:53.8863633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8863999Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8864132Z 2025-11-03T16:39:53.8864229Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8864572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8864877Z res = mod(**inputs) 2025-11-03T16:39:53.8865206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8865567Z outputs = self.model.decoder( 2025-11-03T16:39:53.8865925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8866282Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8866632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8866971Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8867335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8867692Z return func(*args, **kwargs) 2025-11-03T16:39:53.8868045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8868429Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8868808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8869159Z return func(*args, **kwargs) 2025-11-03T16:39:53.8869538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8869959Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8870117Z 2025-11-03T16:39:53.8870217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8870568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8870877Z res = mod(**inputs) 2025-11-03T16:39:53.8871228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8871598Z outputs = self.model.decoder( 2025-11-03T16:39:53.8871961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8872330Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8872698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8873051Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8873411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8873766Z return func(*args, **kwargs) 2025-11-03T16:39:53.8874204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8874650Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8875063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8875448Z return func(*args, **kwargs) 2025-11-03T16:39:53.8875842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8876257Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8876402Z 2025-11-03T16:39:53.8876523Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8876908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8877234Z res = mod(**inputs) 2025-11-03T16:39:53.8877616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8878022Z outputs = self.model.decoder( 2025-11-03T16:39:53.8878425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8878823Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8879190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8879571Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8879969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8880359Z return func(*args, **kwargs) 2025-11-03T16:39:53.8880755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8881181Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8881591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8881972Z return func(*args, **kwargs) 2025-11-03T16:39:53.8882356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8882766Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8882917Z 2025-11-03T16:39:53.8883001Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8883225Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8883471Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8883696Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8884060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8884369Z res = mod(**inputs) 2025-11-03T16:39:53.8884718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8885081Z outputs = self.model.decoder( 2025-11-03T16:39:53.8885444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8885812Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8886146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8886490Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8886863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8887222Z return func(*args, **kwargs) 2025-11-03T16:39:53.8887575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8887966Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8888347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8888713Z return func(*args, **kwargs) 2025-11-03T16:39:53.8889073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8889456Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8889587Z 2025-11-03T16:39:53.8889694Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8890032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8890340Z res = mod(**inputs) 2025-11-03T16:39:53.8890685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8891073Z outputs = self.model.decoder( 2025-11-03T16:39:53.8891426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8891778Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8892108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8892449Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8892809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8893165Z return func(*args, **kwargs) 2025-11-03T16:39:53.8893523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8893957Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8894136Z 2025-11-03T16:39:53.8894238Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8894594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8894892Z res = mod(**inputs) 2025-11-03T16:39:53.8895235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8895599Z outputs = self.model.decoder( 2025-11-03T16:39:53.8895955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8896315Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8896655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8896999Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8897356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8897701Z return func(*args, **kwargs) 2025-11-03T16:39:53.8898050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8898448Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8898822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8899152Z return self.act(input) 2025-11-03T16:39:53.8899257Z 2025-11-03T16:39:53.8899366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8899720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8900031Z res = mod(**inputs) 2025-11-03T16:39:53.8900385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8900746Z outputs = self.model.decoder( 2025-11-03T16:39:53.8901099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8901453Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8901795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8902130Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8902484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8902822Z return func(*args, **kwargs) 2025-11-03T16:39:53.8903171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8903536Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8903664Z 2025-11-03T16:39:53.8903768Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8904099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8904391Z res = mod(**inputs) 2025-11-03T16:39:53.8904723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8905082Z outputs = self.model.decoder( 2025-11-03T16:39:53.8905434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8905792Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8906111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8906447Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8906800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8907181Z return func(*args, **kwargs) 2025-11-03T16:39:53.8907525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8907909Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8908285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8908630Z return func(*args, **kwargs) 2025-11-03T16:39:53.8908985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8909380Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8909545Z 2025-11-03T16:39:53.8909661Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8910001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8910305Z res = mod(**inputs) 2025-11-03T16:39:53.8910656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8911009Z outputs = self.model.decoder( 2025-11-03T16:39:53.8911369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8911722Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8912048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8912387Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8912770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8913130Z return func(*args, **kwargs) 2025-11-03T16:39:53.8913668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8914160Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8914569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8915006Z return func(*args, **kwargs) 2025-11-03T16:39:53.8915449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8915826Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8915960Z 2025-11-03T16:39:53.8916073Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8916432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8916758Z res = mod(**inputs) 2025-11-03T16:39:53.8917126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8917524Z outputs = self.model.decoder( 2025-11-03T16:39:53.8917908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8918300Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8918660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8919030Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8919415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8919784Z return func(*args, **kwargs) 2025-11-03T16:39:53.8920170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8920594Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8921059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8921433Z return func(*args, **kwargs) 2025-11-03T16:39:53.8921804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8922211Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8922360Z 2025-11-03T16:39:53.8922442Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8922661Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8922864Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8923103Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8923494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8923816Z res = mod(**inputs) 2025-11-03T16:39:53.8924173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8924565Z outputs = self.model.decoder( 2025-11-03T16:39:53.8924945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8925303Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8925628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8925960Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8926317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8926665Z return func(*args, **kwargs) 2025-11-03T16:39:53.8927042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8927431Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8927799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8928144Z return func(*args, **kwargs) 2025-11-03T16:39:53.8928493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8928880Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8929009Z 2025-11-03T16:39:53.8929105Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8929441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8929741Z res = mod(**inputs) 2025-11-03T16:39:53.8930080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8930443Z outputs = self.model.decoder( 2025-11-03T16:39:53.8930796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8931159Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8931487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8931830Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8932187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8932526Z return func(*args, **kwargs) 2025-11-03T16:39:53.8932875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8933281Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8933443Z 2025-11-03T16:39:53.8933549Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8933896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8934192Z res = mod(**inputs) 2025-11-03T16:39:53.8934527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8934887Z outputs = self.model.decoder( 2025-11-03T16:39:53.8935243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8935603Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8935933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8936274Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8936640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8936991Z return func(*args, **kwargs) 2025-11-03T16:39:53.8937338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8937739Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8938105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8938425Z return self.act(input) 2025-11-03T16:39:53.8938528Z 2025-11-03T16:39:53.8938626Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8938965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8939265Z res = mod(**inputs) 2025-11-03T16:39:53.8939616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8939976Z outputs = self.model.decoder( 2025-11-03T16:39:53.8940323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8940683Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8941010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8941347Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8941716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8942065Z return func(*args, **kwargs) 2025-11-03T16:39:53.8942428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8942802Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8942931Z 2025-11-03T16:39:53.8943037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8943363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8943668Z res = mod(**inputs) 2025-11-03T16:39:53.8944002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8944367Z outputs = self.model.decoder( 2025-11-03T16:39:53.8944722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8945078Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8945404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8945741Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8946095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8946437Z return func(*args, **kwargs) 2025-11-03T16:39:53.8946786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8947191Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8947563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8947909Z return func(*args, **kwargs) 2025-11-03T16:39:53.8948255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8948654Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8948816Z 2025-11-03T16:39:53.8948914Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8949251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8949573Z res = mod(**inputs) 2025-11-03T16:39:53.8949904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8950270Z outputs = self.model.decoder( 2025-11-03T16:39:53.8950626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8950984Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8951305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8951650Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8952005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8952354Z return func(*args, **kwargs) 2025-11-03T16:39:53.8952718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8953094Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8953465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8953808Z return func(*args, **kwargs) 2025-11-03T16:39:53.8954229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8954659Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8954800Z 2025-11-03T16:39:53.8954905Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8955269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8955599Z res = mod(**inputs) 2025-11-03T16:39:53.8955965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8956320Z outputs = self.model.decoder( 2025-11-03T16:39:53.8956680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8957111Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8957453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8957806Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8958169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8958530Z return func(*args, **kwargs) 2025-11-03T16:39:53.8958890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8959292Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8959673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8960034Z return func(*args, **kwargs) 2025-11-03T16:39:53.8960520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.8960905Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.8961043Z 2025-11-03T16:39:53.8961148Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8961349Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8961553Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.8961779Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8962122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8962430Z res = mod(**inputs) 2025-11-03T16:39:53.8962801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8963188Z outputs = self.model.decoder( 2025-11-03T16:39:53.8963569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8963943Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8964275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8964629Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8964998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8965361Z return func(*args, **kwargs) 2025-11-03T16:39:53.8965721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8966108Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8966510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8966865Z return func(*args, **kwargs) 2025-11-03T16:39:53.8967226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.8967609Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.8967739Z 2025-11-03T16:39:53.8967837Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8968197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8968504Z res = mod(**inputs) 2025-11-03T16:39:53.8968851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8969217Z outputs = self.model.decoder( 2025-11-03T16:39:53.8969586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8969962Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8970299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8970653Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8971009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8971369Z return func(*args, **kwargs) 2025-11-03T16:39:53.8971734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8972152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8972318Z 2025-11-03T16:39:53.8972425Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8972765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8973078Z res = mod(**inputs) 2025-11-03T16:39:53.8973426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8973821Z outputs = self.model.decoder( 2025-11-03T16:39:53.8974176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8974553Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8974893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8975240Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8975602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8975950Z return func(*args, **kwargs) 2025-11-03T16:39:53.8976319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.8976735Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.8977109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.8977431Z return self.act(input) 2025-11-03T16:39:53.8977544Z 2025-11-03T16:39:53.8977643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8977986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8978296Z res = mod(**inputs) 2025-11-03T16:39:53.8978641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8979001Z outputs = self.model.decoder( 2025-11-03T16:39:53.8979379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8979755Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8980097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8980447Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8980804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8981166Z return func(*args, **kwargs) 2025-11-03T16:39:53.8981539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.8981916Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.8982046Z 2025-11-03T16:39:53.8982146Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8982495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8982810Z res = mod(**inputs) 2025-11-03T16:39:53.8983178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8983576Z outputs = self.model.decoder( 2025-11-03T16:39:53.8983939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8984319Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8984663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8985020Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8985375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8985733Z return func(*args, **kwargs) 2025-11-03T16:39:53.8986100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8986517Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8986905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8987293Z return func(*args, **kwargs) 2025-11-03T16:39:53.8987661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.8988084Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.8988252Z 2025-11-03T16:39:53.8988361Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8988714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8989024Z res = mod(**inputs) 2025-11-03T16:39:53.8989381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8989784Z outputs = self.model.decoder( 2025-11-03T16:39:53.8990158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8990541Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8990876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8991233Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8991609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8991976Z return func(*args, **kwargs) 2025-11-03T16:39:53.8992334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8992737Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8993147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8993515Z return func(*args, **kwargs) 2025-11-03T16:39:53.8993891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.8994335Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.8994476Z 2025-11-03T16:39:53.8994576Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.8994928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.8995273Z res = mod(**inputs) 2025-11-03T16:39:53.8995608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.8995974Z outputs = self.model.decoder( 2025-11-03T16:39:53.8996337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.8996700Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.8997033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.8997372Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.8997735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8998086Z return func(*args, **kwargs) 2025-11-03T16:39:53.8998439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.8998830Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.8999198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.8999549Z return func(*args, **kwargs) 2025-11-03T16:39:53.8999904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.9000284Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.9000433Z 2025-11-03T16:39:53.9000509Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9000712Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9000910Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9001130Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9001464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9001763Z res = mod(**inputs) 2025-11-03T16:39:53.9002101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9002462Z outputs = self.model.decoder( 2025-11-03T16:39:53.9002819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9003186Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9003518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9003863Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9004227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9004580Z return func(*args, **kwargs) 2025-11-03T16:39:53.9004929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9005314Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9005690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9006042Z return func(*args, **kwargs) 2025-11-03T16:39:53.9006411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.9006788Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.9006925Z 2025-11-03T16:39:53.9007022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9007360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9007662Z res = mod(**inputs) 2025-11-03T16:39:53.9007997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9008374Z outputs = self.model.decoder( 2025-11-03T16:39:53.9008735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9009097Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9009427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9009760Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9010120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9010470Z return func(*args, **kwargs) 2025-11-03T16:39:53.9010821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9011220Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9011389Z 2025-11-03T16:39:53.9011489Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9011827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9012127Z res = mod(**inputs) 2025-11-03T16:39:53.9012470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9012828Z outputs = self.model.decoder( 2025-11-03T16:39:53.9013181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9013708Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9014041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9014382Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9014734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9015080Z return func(*args, **kwargs) 2025-11-03T16:39:53.9015428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9015828Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9016231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.9016555Z return self.act(input) 2025-11-03T16:39:53.9016667Z 2025-11-03T16:39:53.9016763Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9017097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9017394Z res = mod(**inputs) 2025-11-03T16:39:53.9017725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9018093Z outputs = self.model.decoder( 2025-11-03T16:39:53.9018448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9018809Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9019138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9019496Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9019854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9020199Z return func(*args, **kwargs) 2025-11-03T16:39:53.9020546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.9020906Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.9021074Z 2025-11-03T16:39:53.9021173Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9021510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9021813Z res = mod(**inputs) 2025-11-03T16:39:53.9022148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9022506Z outputs = self.model.decoder( 2025-11-03T16:39:53.9022756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9022825Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9023041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9023119Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9023346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9023421Z return func(*args, **kwargs) 2025-11-03T16:39:53.9023659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9023760Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9023987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9024054Z return func(*args, **kwargs) 2025-11-03T16:39:53.9024295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.9024422Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.9024425Z 2025-11-03T16:39:53.9024531Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9024717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9024786Z res = mod(**inputs) 2025-11-03T16:39:53.9025021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9025087Z outputs = self.model.decoder( 2025-11-03T16:39:53.9025326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9025393Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9025623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9025701Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9025926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9025996Z return func(*args, **kwargs) 2025-11-03T16:39:53.9026232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9026335Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9026558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9026630Z return func(*args, **kwargs) 2025-11-03T16:39:53.9026881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.9026958Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.9026963Z 2025-11-03T16:39:53.9027066Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9027251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9027318Z res = mod(**inputs) 2025-11-03T16:39:53.9027552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9027636Z outputs = self.model.decoder( 2025-11-03T16:39:53.9027884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9027950Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9028165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9028241Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9028469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9028541Z return func(*args, **kwargs) 2025-11-03T16:39:53.9028774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9028874Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9029106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9029175Z return func(*args, **kwargs) 2025-11-03T16:39:53.9029408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.9029490Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.9029493Z 2025-11-03T16:39:53.9029573Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9029648Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9029734Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9029845Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9030028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9030094Z res = mod(**inputs) 2025-11-03T16:39:53.9030328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9030403Z outputs = self.model.decoder( 2025-11-03T16:39:53.9030635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9030704Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9030918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9031007Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9031244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9031309Z return func(*args, **kwargs) 2025-11-03T16:39:53.9031552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9031644Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9031875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9031947Z return func(*args, **kwargs) 2025-11-03T16:39:53.9032185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.9032268Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.9032271Z 2025-11-03T16:39:53.9032420Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9032606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9032676Z res = mod(**inputs) 2025-11-03T16:39:53.9032907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9032982Z outputs = self.model.decoder( 2025-11-03T16:39:53.9033213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9033298Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9033512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9033587Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9033820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9033889Z return func(*args, **kwargs) 2025-11-03T16:39:53.9034207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9034337Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9034341Z 2025-11-03T16:39:53.9034442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9034643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9034708Z res = mod(**inputs) 2025-11-03T16:39:53.9034965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9035037Z outputs = self.model.decoder( 2025-11-03T16:39:53.9035283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9035363Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9035614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9035713Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9035956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9036031Z return func(*args, **kwargs) 2025-11-03T16:39:53.9036280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9036399Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9036619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.9036688Z return self.act(input) 2025-11-03T16:39:53.9036691Z 2025-11-03T16:39:53.9036800Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9037019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9037085Z res = mod(**inputs) 2025-11-03T16:39:53.9037340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9037411Z outputs = self.model.decoder( 2025-11-03T16:39:53.9037666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9037737Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9037956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9038040Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9038292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9038378Z return func(*args, **kwargs) 2025-11-03T16:39:53.9038629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.9038717Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.9038720Z 2025-11-03T16:39:53.9038820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9039015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9039102Z res = mod(**inputs) 2025-11-03T16:39:53.9039353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9039431Z outputs = self.model.decoder( 2025-11-03T16:39:53.9039682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9039751Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9039985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9040064Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9040314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9040383Z return func(*args, **kwargs) 2025-11-03T16:39:53.9040635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9040740Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9040982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9041057Z return func(*args, **kwargs) 2025-11-03T16:39:53.9041311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.9041430Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.9041434Z 2025-11-03T16:39:53.9041537Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9041750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9041820Z res = mod(**inputs) 2025-11-03T16:39:53.9042069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9042149Z outputs = self.model.decoder( 2025-11-03T16:39:53.9042399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9042467Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9042693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9042768Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9043031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9043103Z return func(*args, **kwargs) 2025-11-03T16:39:53.9043356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9043451Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9043692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9043767Z return func(*args, **kwargs) 2025-11-03T16:39:53.9044015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.9044100Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.9044103Z 2025-11-03T16:39:53.9044219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9044417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9044487Z res = mod(**inputs) 2025-11-03T16:39:53.9044739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9044813Z outputs = self.model.decoder( 2025-11-03T16:39:53.9045060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9045140Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9045355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9045427Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9045662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9045728Z return func(*args, **kwargs) 2025-11-03T16:39:53.9045972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9046065Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9046286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9046356Z return func(*args, **kwargs) 2025-11-03T16:39:53.9046590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.9046678Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.9046681Z 2025-11-03T16:39:53.9046755Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9046826Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9046906Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9047002Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9047193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9047268Z res = mod(**inputs) 2025-11-03T16:39:53.9047505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9047583Z outputs = self.model.decoder( 2025-11-03T16:39:53.9047819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9047895Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9048101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9048180Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9048407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9048492Z return func(*args, **kwargs) 2025-11-03T16:39:53.9048736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9048828Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9049060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9049123Z return func(*args, **kwargs) 2025-11-03T16:39:53.9049359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.9049443Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.9049446Z 2025-11-03T16:39:53.9049543Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9049735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9049810Z res = mod(**inputs) 2025-11-03T16:39:53.9050047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9050123Z outputs = self.model.decoder( 2025-11-03T16:39:53.9050355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9050428Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9050634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9050727Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9050950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9051013Z return func(*args, **kwargs) 2025-11-03T16:39:53.9051255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9051368Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9051371Z 2025-11-03T16:39:53.9051475Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9051657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9051717Z res = mod(**inputs) 2025-11-03T16:39:53.9051963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9052032Z outputs = self.model.decoder( 2025-11-03T16:39:53.9052271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9052336Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9052548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9052624Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9052848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9052935Z return func(*args, **kwargs) 2025-11-03T16:39:53.9053174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9053292Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9053498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.9053563Z return self.act(input) 2025-11-03T16:39:53.9053567Z 2025-11-03T16:39:53.9053670Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9053857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9053923Z res = mod(**inputs) 2025-11-03T16:39:53.9054178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9054247Z outputs = self.model.decoder( 2025-11-03T16:39:53.9054495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9054562Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9054775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9054851Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9055084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9055148Z return func(*args, **kwargs) 2025-11-03T16:39:53.9055381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.9055480Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.9055484Z 2025-11-03T16:39:53.9055579Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9055769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9055829Z res = mod(**inputs) 2025-11-03T16:39:53.9056063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9056139Z outputs = self.model.decoder( 2025-11-03T16:39:53.9056389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9056462Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9056671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9056744Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9056980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9057044Z return func(*args, **kwargs) 2025-11-03T16:39:53.9057289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9057380Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9057611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9057676Z return func(*args, **kwargs) 2025-11-03T16:39:53.9057916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 201, in forward 2025-11-03T16:39:53.9058028Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:39:53.9058032Z 2025-11-03T16:39:53.9058129Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9058324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9058384Z res = mod(**inputs) 2025-11-03T16:39:53.9058643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9058717Z outputs = self.model.decoder( 2025-11-03T16:39:53.9058954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9059031Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9059240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9059324Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9059553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9059621Z return func(*args, **kwargs) 2025-11-03T16:39:53.9059876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9059973Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9060202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9060265Z return func(*args, **kwargs) 2025-11-03T16:39:53.9060497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 220, in forward 2025-11-03T16:39:53.9060582Z key_states = self.k_proj(current_states) 2025-11-03T16:39:53.9060585Z 2025-11-03T16:39:53.9060682Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9060873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9060932Z res = mod(**inputs) 2025-11-03T16:39:53.9061181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9061259Z outputs = self.model.decoder( 2025-11-03T16:39:53.9061492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9061566Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9061773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9061867Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9062092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9062157Z return func(*args, **kwargs) 2025-11-03T16:39:53.9062404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9062497Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9062735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9062802Z return func(*args, **kwargs) 2025-11-03T16:39:53.9063038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 221, in forward 2025-11-03T16:39:53.9063129Z value_states = self.v_proj(current_states) 2025-11-03T16:39:53.9063132Z 2025-11-03T16:39:53.9063207Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9063287Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9063360Z cudagraph partition due to non gpu ops 2025-11-03T16:39:53.9063455Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9063649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9063709Z res = mod(**inputs) 2025-11-03T16:39:53.9063956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9064024Z outputs = self.model.decoder( 2025-11-03T16:39:53.9064286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9064353Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9064556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9064638Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9064863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9064933Z return func(*args, **kwargs) 2025-11-03T16:39:53.9065166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 370, in forward 2025-11-03T16:39:53.9065275Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:39:53.9065508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9065573Z return func(*args, **kwargs) 2025-11-03T16:39:53.9065814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 293, in forward 2025-11-03T16:39:53.9065890Z attn_output = self.out_proj(attn_output) 2025-11-03T16:39:53.9065895Z 2025-11-03T16:39:53.9065996Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9066179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9066239Z res = mod(**inputs) 2025-11-03T16:39:53.9066480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9066547Z outputs = self.model.decoder( 2025-11-03T16:39:53.9066801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9066870Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9067075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9067155Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9067380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9067492Z return func(*args, **kwargs) 2025-11-03T16:39:53.9067724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9067835Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9067847Z 2025-11-03T16:39:53.9067943Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9068128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9068195Z res = mod(**inputs) 2025-11-03T16:39:53.9068431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9068504Z outputs = self.model.decoder( 2025-11-03T16:39:53.9068738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9068805Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9069017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9069088Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9069321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9069385Z return func(*args, **kwargs) 2025-11-03T16:39:53.9069618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 404, in forward 2025-11-03T16:39:53.9069752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:39:53.9069952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:39:53.9070025Z return self.act(input) 2025-11-03T16:39:53.9070028Z 2025-11-03T16:39:53.9070123Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9070315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9070383Z res = mod(**inputs) 2025-11-03T16:39:53.9070630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 827, in forward 2025-11-03T16:39:53.9070708Z outputs = self.model.decoder( 2025-11-03T16:39:53.9070965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 651, in forward 2025-11-03T16:39:53.9071042Z layer_outputs = decoder_layer( 2025-11-03T16:39:53.9071254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:39:53.9071329Z return super().__call__(*args, **kwargs) 2025-11-03T16:39:53.9071568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:39:53.9071638Z return func(*args, **kwargs) 2025-11-03T16:39:53.9071890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 406, in forward 2025-11-03T16:39:53.9071968Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:39:53.9071971Z 2025-11-03T16:39:53.9072069Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9072279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9072343Z res = mod(**inputs) 2025-11-03T16:39:53.9072588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 843, in forward 2025-11-03T16:39:53.9072680Z logits = self.output_projection(outputs[0]) 2025-11-03T16:39:53.9072683Z 2025-11-03T16:39:53.9072785Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:39:53.9072970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:39:53.9073046Z res = mod(**inputs) 2025-11-03T16:39:53.9073295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 848, in forward 2025-11-03T16:39:53.9073436Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:39:53.9073440Z 2025-11-03T16:40:03.3914491Z Compilation time (from dynamo_timed): 15.940334617 2025-11-03T16:40:03.3952647Z pass 2025-11-03T16:40:03.3957171Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:03.3960976Z TIMING: _recursive_pre_grad_passes:0.00704 _recursive_joint_graph_passes:0.7012 _recursive_post_grad_passes:0.06631 async_compile.wait:0.78098 code_gen:8.61248 inductor_compile:9.84958 backend_compile:13.07565 gc:0.00113 entire_frame_compile:15.94033 total_wall_time:15.94033 2025-11-03T16:40:03.3962205Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:8236 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:2504 2025-11-03T16:40:03.3962710Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-11-03T16:40:05.5871385Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:40:05.5872251Z import pynvml # type: ignore[import] 2025-11-03T16:40:08.6994784Z 2025-11-03T16:40:13.8645364Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:40:13.8646590Z loading model: 0it [00:05, ?it/s] 2025-11-03T16:40:13.8660435Z cpu eval XGLMForCausalLM 2025-11-03T16:40:14.2886027Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-11-03T16:40:14.3791405Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:14.8635293Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:15.3742225Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:28.9713836Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9718634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9724260Z res = mod(**inputs) 2025-11-03T16:40:28.9729038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9733483Z outputs = self.model( 2025-11-03T16:40:28.9737934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9741897Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9744340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9744850Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9750209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9756110Z return func(*args, **kwargs) 2025-11-03T16:40:28.9758893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9764677Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9767180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9773259Z return func(*args, **kwargs) 2025-11-03T16:40:28.9788712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:28.9790745Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:28.9795384Z 2025-11-03T16:40:28.9799357Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9804305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9809040Z res = mod(**inputs) 2025-11-03T16:40:28.9811557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9811993Z outputs = self.model( 2025-11-03T16:40:28.9812369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9812765Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9813130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9813713Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9814097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9814472Z return func(*args, **kwargs) 2025-11-03T16:40:28.9814838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9815251Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9815652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9816034Z return func(*args, **kwargs) 2025-11-03T16:40:28.9816553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:28.9816938Z key_states = self.k_proj(current_states) 2025-11-03T16:40:28.9817075Z 2025-11-03T16:40:28.9817192Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9817545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9817861Z res = mod(**inputs) 2025-11-03T16:40:28.9818216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9818585Z outputs = self.model( 2025-11-03T16:40:28.9818930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9819335Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9819688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9820053Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9820434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9820804Z return func(*args, **kwargs) 2025-11-03T16:40:28.9821188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9821610Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9822012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9822373Z return func(*args, **kwargs) 2025-11-03T16:40:28.9822759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:28.9823178Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:28.9823353Z 2025-11-03T16:40:28.9823456Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9823943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9824258Z res = mod(**inputs) 2025-11-03T16:40:28.9824625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9824990Z outputs = self.model( 2025-11-03T16:40:28.9825350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9825720Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9826054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9826409Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9826785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9827145Z return func(*args, **kwargs) 2025-11-03T16:40:28.9827506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9827903Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9828302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9828694Z return func(*args, **kwargs) 2025-11-03T16:40:28.9829069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:28.9829523Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:28.9829716Z 2025-11-03T16:40:28.9829820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9830213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9830529Z res = mod(**inputs) 2025-11-03T16:40:28.9830882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9831257Z outputs = self.model( 2025-11-03T16:40:28.9831606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9832041Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9832386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9832747Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9833158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9833531Z return func(*args, **kwargs) 2025-11-03T16:40:28.9833909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9834409Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9834821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9835208Z return func(*args, **kwargs) 2025-11-03T16:40:28.9835639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:28.9836055Z value_states = self.v_proj(current_states) 2025-11-03T16:40:28.9836197Z 2025-11-03T16:40:28.9836309Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9836684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9837000Z res = mod(**inputs) 2025-11-03T16:40:28.9837353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9837730Z outputs = self.model( 2025-11-03T16:40:28.9838090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9838463Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9838827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9839200Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9839581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9839956Z return func(*args, **kwargs) 2025-11-03T16:40:28.9840321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9840728Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9841124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9841494Z return func(*args, **kwargs) 2025-11-03T16:40:28.9841856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:28.9842252Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:28.9842406Z 2025-11-03T16:40:28.9842507Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9842853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9843170Z res = mod(**inputs) 2025-11-03T16:40:28.9843512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9843883Z outputs = self.model( 2025-11-03T16:40:28.9844234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9844634Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9844995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9845352Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9845732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9846096Z return func(*args, **kwargs) 2025-11-03T16:40:28.9846461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9846865Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9847294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9847667Z return func(*args, **kwargs) 2025-11-03T16:40:28.9848032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:28.9848469Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:28.9848640Z 2025-11-03T16:40:28.9848740Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9849095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9849425Z res = mod(**inputs) 2025-11-03T16:40:28.9849777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9850140Z outputs = self.model( 2025-11-03T16:40:28.9850491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9850861Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9851197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9851550Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9851917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9852281Z return func(*args, **kwargs) 2025-11-03T16:40:28.9852629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9853020Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9853400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9853749Z return func(*args, **kwargs) 2025-11-03T16:40:28.9854102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:28.9854479Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:28.9854613Z 2025-11-03T16:40:28.9854720Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9855060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9855360Z res = mod(**inputs) 2025-11-03T16:40:28.9855701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9856058Z outputs = self.model( 2025-11-03T16:40:28.9856396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9856761Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9857089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9857432Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9857817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9858184Z return func(*args, **kwargs) 2025-11-03T16:40:28.9858536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:28.9858944Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:28.9859118Z 2025-11-03T16:40:28.9859217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9859556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9859863Z res = mod(**inputs) 2025-11-03T16:40:28.9860207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9860575Z outputs = self.model( 2025-11-03T16:40:28.9860922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9861290Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9861629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9861971Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9862335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9862690Z return func(*args, **kwargs) 2025-11-03T16:40:28.9863043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:28.9863449Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:28.9863831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:28.9864164Z return self.act(input) 2025-11-03T16:40:28.9864277Z 2025-11-03T16:40:28.9864377Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9864717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9865021Z res = mod(**inputs) 2025-11-03T16:40:28.9865357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9865737Z outputs = self.model( 2025-11-03T16:40:28.9866082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9866451Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9866782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9867131Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9867500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9867859Z return func(*args, **kwargs) 2025-11-03T16:40:28.9868218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:28.9868586Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:28.9868725Z 2025-11-03T16:40:28.9868825Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9869178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9869493Z res = mod(**inputs) 2025-11-03T16:40:28.9869841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9870207Z outputs = self.model( 2025-11-03T16:40:28.9870565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9871625Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9871972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9872322Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9872696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9873064Z return func(*args, **kwargs) 2025-11-03T16:40:28.9873429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9873827Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9874316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9874760Z return func(*args, **kwargs) 2025-11-03T16:40:28.9875160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:28.9875621Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:28.9875791Z 2025-11-03T16:40:28.9875908Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9876263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9876593Z res = mod(**inputs) 2025-11-03T16:40:28.9876962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9877334Z outputs = self.model( 2025-11-03T16:40:28.9877680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9878071Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9878419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9878779Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9879151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9879514Z return func(*args, **kwargs) 2025-11-03T16:40:28.9879873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9880295Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9880689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9881048Z return func(*args, **kwargs) 2025-11-03T16:40:28.9881411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:28.9881796Z key_states = self.k_proj(current_states) 2025-11-03T16:40:28.9881929Z 2025-11-03T16:40:28.9882039Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9882389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9882700Z res = mod(**inputs) 2025-11-03T16:40:28.9883052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9883430Z outputs = self.model( 2025-11-03T16:40:28.9883790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9884182Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9884515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9884873Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9885252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9885637Z return func(*args, **kwargs) 2025-11-03T16:40:28.9885993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9886395Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9886787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9887156Z return func(*args, **kwargs) 2025-11-03T16:40:28.9887517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:28.9887925Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:28.9888094Z 2025-11-03T16:40:28.9888199Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9888568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9888889Z res = mod(**inputs) 2025-11-03T16:40:28.9889250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9889606Z outputs = self.model( 2025-11-03T16:40:28.9889950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9890314Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9890647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9890989Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9891355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9891729Z return func(*args, **kwargs) 2025-11-03T16:40:28.9892084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9892473Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9892845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9893197Z return func(*args, **kwargs) 2025-11-03T16:40:28.9893571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:28.9894010Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:28.9894195Z 2025-11-03T16:40:28.9894305Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9894646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9894967Z res = mod(**inputs) 2025-11-03T16:40:28.9895316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9895685Z outputs = self.model( 2025-11-03T16:40:28.9896029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9896399Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9896739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9897095Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9897465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9897821Z return func(*args, **kwargs) 2025-11-03T16:40:28.9898180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9898574Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9898965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9899340Z return func(*args, **kwargs) 2025-11-03T16:40:28.9899689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:28.9900066Z value_states = self.v_proj(current_states) 2025-11-03T16:40:28.9900203Z 2025-11-03T16:40:28.9900309Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9900650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9900951Z res = mod(**inputs) 2025-11-03T16:40:28.9901289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9901648Z outputs = self.model( 2025-11-03T16:40:28.9902006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9902374Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9902702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9903049Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9903413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9903775Z return func(*args, **kwargs) 2025-11-03T16:40:28.9904163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9904558Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9904955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9905326Z return func(*args, **kwargs) 2025-11-03T16:40:28.9905671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:28.9906047Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:28.9906199Z 2025-11-03T16:40:28.9906296Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9906639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9906962Z res = mod(**inputs) 2025-11-03T16:40:28.9907302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9907656Z outputs = self.model( 2025-11-03T16:40:28.9908001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9908370Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9908705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9909049Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9909418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9909778Z return func(*args, **kwargs) 2025-11-03T16:40:28.9910129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9910518Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9910892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9911262Z return func(*args, **kwargs) 2025-11-03T16:40:28.9911626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:28.9912058Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:28.9912250Z 2025-11-03T16:40:28.9912358Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9912704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9913018Z res = mod(**inputs) 2025-11-03T16:40:28.9913518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9913904Z outputs = self.model( 2025-11-03T16:40:28.9914304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9914695Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9915049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9915457Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9915832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9916173Z return func(*args, **kwargs) 2025-11-03T16:40:28.9916516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9916900Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9917276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9917625Z return func(*args, **kwargs) 2025-11-03T16:40:28.9917960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:28.9918328Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:28.9918457Z 2025-11-03T16:40:28.9918590Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9918932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9919229Z res = mod(**inputs) 2025-11-03T16:40:28.9919562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9919919Z outputs = self.model( 2025-11-03T16:40:28.9920255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9920636Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9920961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9921306Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9921670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9922021Z return func(*args, **kwargs) 2025-11-03T16:40:28.9922360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:28.9922768Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:28.9922939Z 2025-11-03T16:40:28.9923037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9923371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9923673Z res = mod(**inputs) 2025-11-03T16:40:28.9924006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9924356Z outputs = self.model( 2025-11-03T16:40:28.9924693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9925051Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9925379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9925736Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9926087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9926435Z return func(*args, **kwargs) 2025-11-03T16:40:28.9926778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:28.9927172Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:28.9927538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:28.9927857Z return self.act(input) 2025-11-03T16:40:28.9927960Z 2025-11-03T16:40:28.9928064Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9928412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9928710Z res = mod(**inputs) 2025-11-03T16:40:28.9929045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9929398Z outputs = self.model( 2025-11-03T16:40:28.9929735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9930088Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9930414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9930754Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9931110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9931475Z return func(*args, **kwargs) 2025-11-03T16:40:28.9931814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:28.9932183Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:28.9932317Z 2025-11-03T16:40:28.9932413Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9932744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9933045Z res = mod(**inputs) 2025-11-03T16:40:28.9933388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9933740Z outputs = self.model( 2025-11-03T16:40:28.9934075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9934433Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9934752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9935092Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9935449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9935798Z return func(*args, **kwargs) 2025-11-03T16:40:28.9936142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9936524Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9936902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9937254Z return func(*args, **kwargs) 2025-11-03T16:40:28.9937599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:28.9937996Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:28.9938153Z 2025-11-03T16:40:28.9938252Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9938606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9938903Z res = mod(**inputs) 2025-11-03T16:40:28.9939234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9939576Z outputs = self.model( 2025-11-03T16:40:28.9939910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9940264Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9940589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9940926Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9941288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9941639Z return func(*args, **kwargs) 2025-11-03T16:40:28.9941982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9942361Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9942734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9943076Z return func(*args, **kwargs) 2025-11-03T16:40:28.9943420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:28.9943787Z key_states = self.k_proj(current_states) 2025-11-03T16:40:28.9943913Z 2025-11-03T16:40:28.9944015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9944356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9944657Z res = mod(**inputs) 2025-11-03T16:40:28.9944983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9945338Z outputs = self.model( 2025-11-03T16:40:28.9945675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9946022Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9946373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9946719Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9947079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9947427Z return func(*args, **kwargs) 2025-11-03T16:40:28.9947779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9948166Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9948547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9948900Z return func(*args, **kwargs) 2025-11-03T16:40:28.9949249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:28.9949664Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:28.9949831Z 2025-11-03T16:40:28.9949933Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9950281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9950595Z res = mod(**inputs) 2025-11-03T16:40:28.9950941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9951312Z outputs = self.model( 2025-11-03T16:40:28.9951663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9952050Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9952383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9952748Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9953121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9953486Z return func(*args, **kwargs) 2025-11-03T16:40:28.9953878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9954348Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9954771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9955150Z return func(*args, **kwargs) 2025-11-03T16:40:28.9955521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:28.9955968Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:28.9956150Z 2025-11-03T16:40:28.9956250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9956597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9956908Z res = mod(**inputs) 2025-11-03T16:40:28.9957251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9957610Z outputs = self.model( 2025-11-03T16:40:28.9957977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9958342Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9958678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9959026Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9959385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9959760Z return func(*args, **kwargs) 2025-11-03T16:40:28.9960114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9960507Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9960896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9961246Z return func(*args, **kwargs) 2025-11-03T16:40:28.9961600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:28.9961980Z value_states = self.v_proj(current_states) 2025-11-03T16:40:28.9962114Z 2025-11-03T16:40:28.9962219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9962555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9962857Z res = mod(**inputs) 2025-11-03T16:40:28.9963193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9963552Z outputs = self.model( 2025-11-03T16:40:28.9963898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9964257Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9964591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9964939Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9965328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9965677Z return func(*args, **kwargs) 2025-11-03T16:40:28.9966028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9966418Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9966802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9967160Z return func(*args, **kwargs) 2025-11-03T16:40:28.9967509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:28.9967900Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:28.9968068Z 2025-11-03T16:40:28.9968169Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9968517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9968827Z res = mod(**inputs) 2025-11-03T16:40:28.9969161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9969527Z outputs = self.model( 2025-11-03T16:40:28.9969875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9970242Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9970566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9970922Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9971308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9971670Z return func(*args, **kwargs) 2025-11-03T16:40:28.9972030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9972416Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9972806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9973175Z return func(*args, **kwargs) 2025-11-03T16:40:28.9973529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:28.9973948Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:28.9974116Z 2025-11-03T16:40:28.9974215Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9974560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9974866Z res = mod(**inputs) 2025-11-03T16:40:28.9975204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9975557Z outputs = self.model( 2025-11-03T16:40:28.9975899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9976266Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9976601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9976951Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9977307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9977660Z return func(*args, **kwargs) 2025-11-03T16:40:28.9978018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:28.9978437Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:28.9978829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9979189Z return func(*args, **kwargs) 2025-11-03T16:40:28.9979553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:28.9979940Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:28.9980074Z 2025-11-03T16:40:28.9980180Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9980523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9980840Z res = mod(**inputs) 2025-11-03T16:40:28.9981204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9981575Z outputs = self.model( 2025-11-03T16:40:28.9981928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9982299Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9982644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9983008Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9983385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9983751Z return func(*args, **kwargs) 2025-11-03T16:40:28.9984106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:28.9984544Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:28.9984726Z 2025-11-03T16:40:28.9984829Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9985189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9985496Z res = mod(**inputs) 2025-11-03T16:40:28.9985845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9986215Z outputs = self.model( 2025-11-03T16:40:28.9986585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9986963Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9987298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9987659Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9988040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9988412Z return func(*args, **kwargs) 2025-11-03T16:40:28.9988768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:28.9989189Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:28.9989573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:28.9989915Z return self.act(input) 2025-11-03T16:40:28.9990026Z 2025-11-03T16:40:28.9990135Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9990481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9990805Z res = mod(**inputs) 2025-11-03T16:40:28.9991156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9991530Z outputs = self.model( 2025-11-03T16:40:28.9991886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9992278Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9992620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9992983Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9993365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9993715Z return func(*args, **kwargs) 2025-11-03T16:40:28.9994068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:28.9994524Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:28.9994662Z 2025-11-03T16:40:28.9994776Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9995150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9995459Z res = mod(**inputs) 2025-11-03T16:40:28.9995810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:28.9996179Z outputs = self.model( 2025-11-03T16:40:28.9996522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:28.9996883Z layer_outputs = decoder_layer( 2025-11-03T16:40:28.9997217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:28.9997571Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:28.9997972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:28.9998334Z return func(*args, **kwargs) 2025-11-03T16:40:28.9998682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:28.9999057Z hidden_states = residual + hidden_states 2025-11-03T16:40:28.9999196Z 2025-11-03T16:40:28.9999294Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:28.9999635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:28.9999962Z res = mod(**inputs) 2025-11-03T16:40:29.0000301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0000665Z outputs = self.model( 2025-11-03T16:40:29.0001012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0001378Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0001709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0002062Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0002425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0002789Z return func(*args, **kwargs) 2025-11-03T16:40:29.0003144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0003535Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0003920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0004277Z return func(*args, **kwargs) 2025-11-03T16:40:29.0004631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0005034Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0005194Z 2025-11-03T16:40:29.0005313Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0005659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0005970Z res = mod(**inputs) 2025-11-03T16:40:29.0006313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0006671Z outputs = self.model( 2025-11-03T16:40:29.0007016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0007386Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0007721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0008074Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0008453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0008803Z return func(*args, **kwargs) 2025-11-03T16:40:29.0009148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0009528Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0009899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0010240Z return func(*args, **kwargs) 2025-11-03T16:40:29.0010581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0010942Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0011067Z 2025-11-03T16:40:29.0011169Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0011510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0011808Z res = mod(**inputs) 2025-11-03T16:40:29.0012143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0012494Z outputs = self.model( 2025-11-03T16:40:29.0012830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0013348Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0013696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0014054Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0014420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0014778Z return func(*args, **kwargs) 2025-11-03T16:40:29.0015137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0015540Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0015919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0016273Z return func(*args, **kwargs) 2025-11-03T16:40:29.0016611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0017012Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0017171Z 2025-11-03T16:40:29.0017270Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0017609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0017910Z res = mod(**inputs) 2025-11-03T16:40:29.0018241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0018632Z outputs = self.model( 2025-11-03T16:40:29.0018968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0019324Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0019644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0019984Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0020336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0020690Z return func(*args, **kwargs) 2025-11-03T16:40:29.0021038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0021457Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0021839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0022196Z return func(*args, **kwargs) 2025-11-03T16:40:29.0022544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0022965Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0023143Z 2025-11-03T16:40:29.0023242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0023581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0023884Z res = mod(**inputs) 2025-11-03T16:40:29.0024220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0024594Z outputs = self.model( 2025-11-03T16:40:29.0024937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0025294Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0025620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0025962Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0026310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0026693Z return func(*args, **kwargs) 2025-11-03T16:40:29.0027036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0027414Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0027783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0028121Z return func(*args, **kwargs) 2025-11-03T16:40:29.0028463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0028829Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0028962Z 2025-11-03T16:40:29.0029064Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0029390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0029694Z res = mod(**inputs) 2025-11-03T16:40:29.0030025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0030375Z outputs = self.model( 2025-11-03T16:40:29.0030710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0031065Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0031400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0031766Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0032134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0032486Z return func(*args, **kwargs) 2025-11-03T16:40:29.0032842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0033236Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0033622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0033982Z return func(*args, **kwargs) 2025-11-03T16:40:29.0034386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0034814Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0034971Z 2025-11-03T16:40:29.0035075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0035433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0035747Z res = mod(**inputs) 2025-11-03T16:40:29.0036085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0036458Z outputs = self.model( 2025-11-03T16:40:29.0036811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0037193Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0037525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0037896Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0038278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0038643Z return func(*args, **kwargs) 2025-11-03T16:40:29.0038998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0039389Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0039775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0040148Z return func(*args, **kwargs) 2025-11-03T16:40:29.0040501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0040917Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0041086Z 2025-11-03T16:40:29.0041186Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0041527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0041835Z res = mod(**inputs) 2025-11-03T16:40:29.0042175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0042528Z outputs = self.model( 2025-11-03T16:40:29.0042875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0043239Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0043571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0043915Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0044272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0044630Z return func(*args, **kwargs) 2025-11-03T16:40:29.0044979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0045385Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0045768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0046117Z return func(*args, **kwargs) 2025-11-03T16:40:29.0046476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0046845Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0046973Z 2025-11-03T16:40:29.0047077Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0047403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0047704Z res = mod(**inputs) 2025-11-03T16:40:29.0048070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0048426Z outputs = self.model( 2025-11-03T16:40:29.0048764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0049113Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0049439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0049781Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0050141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0050488Z return func(*args, **kwargs) 2025-11-03T16:40:29.0050824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0051237Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0051413Z 2025-11-03T16:40:29.0051510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0051844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0052143Z res = mod(**inputs) 2025-11-03T16:40:29.0052494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0052873Z outputs = self.model( 2025-11-03T16:40:29.0053224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0053632Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0053956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0054303Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0054664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0055019Z return func(*args, **kwargs) 2025-11-03T16:40:29.0055374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0055773Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0056139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0056465Z return self.act(input) 2025-11-03T16:40:29.0056572Z 2025-11-03T16:40:29.0056678Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0057010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0057318Z res = mod(**inputs) 2025-11-03T16:40:29.0057658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0058019Z outputs = self.model( 2025-11-03T16:40:29.0058385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0058738Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0059068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0059414Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0059777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0060130Z return func(*args, **kwargs) 2025-11-03T16:40:29.0060483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0060860Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0060991Z 2025-11-03T16:40:29.0061112Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0061455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0061758Z res = mod(**inputs) 2025-11-03T16:40:29.0062097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0062456Z outputs = self.model( 2025-11-03T16:40:29.0062798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0063159Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0063495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0063844Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0064225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0064582Z return func(*args, **kwargs) 2025-11-03T16:40:29.0064931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0065321Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0065704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0066080Z return func(*args, **kwargs) 2025-11-03T16:40:29.0066434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0066829Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0066996Z 2025-11-03T16:40:29.0067096Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0067438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0067745Z res = mod(**inputs) 2025-11-03T16:40:29.0068075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0068436Z outputs = self.model( 2025-11-03T16:40:29.0068781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0069153Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0069501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0069849Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0070223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0070588Z return func(*args, **kwargs) 2025-11-03T16:40:29.0070952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0071355Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0071755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0072123Z return func(*args, **kwargs) 2025-11-03T16:40:29.0072490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0072879Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0073014Z 2025-11-03T16:40:29.0073117Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0073467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0073787Z res = mod(**inputs) 2025-11-03T16:40:29.0074214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0074617Z outputs = self.model( 2025-11-03T16:40:29.0074978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0075368Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0075729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0076090Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0076467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0076826Z return func(*args, **kwargs) 2025-11-03T16:40:29.0077191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0077596Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0078009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0078369Z return func(*args, **kwargs) 2025-11-03T16:40:29.0078731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0079145Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0079307Z 2025-11-03T16:40:29.0079415Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0079784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0080092Z res = mod(**inputs) 2025-11-03T16:40:29.0080443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0080813Z outputs = self.model( 2025-11-03T16:40:29.0081167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0081541Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0081873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0082234Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0082609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0082984Z return func(*args, **kwargs) 2025-11-03T16:40:29.0083340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0083744Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0084134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0084502Z return func(*args, **kwargs) 2025-11-03T16:40:29.0084871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0085320Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0085512Z 2025-11-03T16:40:29.0085613Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0085964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0086278Z res = mod(**inputs) 2025-11-03T16:40:29.0086624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0086995Z outputs = self.model( 2025-11-03T16:40:29.0087348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0087722Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0088085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0088439Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0088816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0089184Z return func(*args, **kwargs) 2025-11-03T16:40:29.0089545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0089946Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0090330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0090694Z return func(*args, **kwargs) 2025-11-03T16:40:29.0091110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0091497Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0091633Z 2025-11-03T16:40:29.0091739Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0092066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0092370Z res = mod(**inputs) 2025-11-03T16:40:29.0092702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0093051Z outputs = self.model( 2025-11-03T16:40:29.0093397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0093752Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0094078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0094426Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0094794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0095132Z return func(*args, **kwargs) 2025-11-03T16:40:29.0095475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0095857Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0096225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0096563Z return func(*args, **kwargs) 2025-11-03T16:40:29.0096907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0097282Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0097422Z 2025-11-03T16:40:29.0097527Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0097869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0098170Z res = mod(**inputs) 2025-11-03T16:40:29.0098509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0098897Z outputs = self.model( 2025-11-03T16:40:29.0099238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0099603Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0099932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0100281Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0100643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0101008Z return func(*args, **kwargs) 2025-11-03T16:40:29.0101358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0101739Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0102109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0102454Z return func(*args, **kwargs) 2025-11-03T16:40:29.0102793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0103193Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0103367Z 2025-11-03T16:40:29.0103464Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0103795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0104095Z res = mod(**inputs) 2025-11-03T16:40:29.0104441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0104789Z outputs = self.model( 2025-11-03T16:40:29.0105136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0105500Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0105835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0106196Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0106555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0106914Z return func(*args, **kwargs) 2025-11-03T16:40:29.0107263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0107662Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0108024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0108379Z return func(*args, **kwargs) 2025-11-03T16:40:29.0108729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0109101Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0109231Z 2025-11-03T16:40:29.0109334Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0109672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0109975Z res = mod(**inputs) 2025-11-03T16:40:29.0110313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0110680Z outputs = self.model( 2025-11-03T16:40:29.0111028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0111402Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0111766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0112119Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0112494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0112847Z return func(*args, **kwargs) 2025-11-03T16:40:29.0113331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0113762Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0113934Z 2025-11-03T16:40:29.0114046Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0114447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0114812Z res = mod(**inputs) 2025-11-03T16:40:29.0115181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0115571Z outputs = self.model( 2025-11-03T16:40:29.0115925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0116287Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0116627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0116988Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0117365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0117729Z return func(*args, **kwargs) 2025-11-03T16:40:29.0118107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0118524Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0118903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0119234Z return self.act(input) 2025-11-03T16:40:29.0119340Z 2025-11-03T16:40:29.0119441Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0119786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0120117Z res = mod(**inputs) 2025-11-03T16:40:29.0120456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0120823Z outputs = self.model( 2025-11-03T16:40:29.0121161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0121527Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0121862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0122212Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0122565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0122923Z return func(*args, **kwargs) 2025-11-03T16:40:29.0123272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0123646Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0123775Z 2025-11-03T16:40:29.0123881Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0124214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0124519Z res = mod(**inputs) 2025-11-03T16:40:29.0124857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0125249Z outputs = self.model( 2025-11-03T16:40:29.0125583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0125929Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0126253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0126593Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0126944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0127281Z return func(*args, **kwargs) 2025-11-03T16:40:29.0127622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0128005Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0128134Z 2025-11-03T16:40:29.0128239Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0128576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0128871Z res = mod(**inputs) 2025-11-03T16:40:29.0129203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0129562Z outputs = self.model( 2025-11-03T16:40:29.0129903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0130254Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0130584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0130924Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0131301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0131658Z return func(*args, **kwargs) 2025-11-03T16:40:29.0132011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0132407Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0132793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0133158Z return func(*args, **kwargs) 2025-11-03T16:40:29.0133500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0133885Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0134049Z 2025-11-03T16:40:29.0134150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0134508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0134805Z res = mod(**inputs) 2025-11-03T16:40:29.0135128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0135477Z outputs = self.model( 2025-11-03T16:40:29.0135812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0136228Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0136561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0136901Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0137261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0137616Z return func(*args, **kwargs) 2025-11-03T16:40:29.0138481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0138896Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0139285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0139647Z return func(*args, **kwargs) 2025-11-03T16:40:29.0140006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0140385Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0140517Z 2025-11-03T16:40:29.0140626Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0140967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0141283Z res = mod(**inputs) 2025-11-03T16:40:29.0141646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0142008Z outputs = self.model( 2025-11-03T16:40:29.0142345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0142711Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0143044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0143394Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0143757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0144109Z return func(*args, **kwargs) 2025-11-03T16:40:29.0144463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0144872Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0145259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0145612Z return func(*args, **kwargs) 2025-11-03T16:40:29.0145967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0146373Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0146532Z 2025-11-03T16:40:29.0146671Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0147012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0147310Z res = mod(**inputs) 2025-11-03T16:40:29.0147648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0148010Z outputs = self.model( 2025-11-03T16:40:29.0148358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0148725Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0149053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0149402Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0149765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0150130Z return func(*args, **kwargs) 2025-11-03T16:40:29.0150474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0150866Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0151250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0151613Z return func(*args, **kwargs) 2025-11-03T16:40:29.0151969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0152410Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0152600Z 2025-11-03T16:40:29.0152703Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0153052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0153365Z res = mod(**inputs) 2025-11-03T16:40:29.0153710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0154077Z outputs = self.model( 2025-11-03T16:40:29.0154509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0154895Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0155262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0155619Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0155985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0156343Z return func(*args, **kwargs) 2025-11-03T16:40:29.0156700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0157091Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0157466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0157821Z return func(*args, **kwargs) 2025-11-03T16:40:29.0158184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0158570Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0158708Z 2025-11-03T16:40:29.0158815Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0159150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0159455Z res = mod(**inputs) 2025-11-03T16:40:29.0159796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0160179Z outputs = self.model( 2025-11-03T16:40:29.0160520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0160887Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0161225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0161578Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0161948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0162300Z return func(*args, **kwargs) 2025-11-03T16:40:29.0162655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0163051Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0163437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0163794Z return func(*args, **kwargs) 2025-11-03T16:40:29.0164142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0164531Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0164674Z 2025-11-03T16:40:29.0164780Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0165124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0165423Z res = mod(**inputs) 2025-11-03T16:40:29.0165781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0166148Z outputs = self.model( 2025-11-03T16:40:29.0166496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0166867Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0167195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0167545Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0167906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0168267Z return func(*args, **kwargs) 2025-11-03T16:40:29.0168629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0168735Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0168967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0169042Z return func(*args, **kwargs) 2025-11-03T16:40:29.0169278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0169410Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0169413Z 2025-11-03T16:40:29.0169510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0169698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0169768Z res = mod(**inputs) 2025-11-03T16:40:29.0170023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0170097Z outputs = self.model( 2025-11-03T16:40:29.0170334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0170402Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0170620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0170715Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0170953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0171019Z return func(*args, **kwargs) 2025-11-03T16:40:29.0171263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0171360Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0171591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0171670Z return func(*args, **kwargs) 2025-11-03T16:40:29.0171907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0171992Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0171996Z 2025-11-03T16:40:29.0172096Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0172286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0172354Z res = mod(**inputs) 2025-11-03T16:40:29.0172594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0172666Z outputs = self.model( 2025-11-03T16:40:29.0172906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0172991Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0173210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0173286Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0173536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0173604Z return func(*args, **kwargs) 2025-11-03T16:40:29.0173856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0173976Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0173980Z 2025-11-03T16:40:29.0174081Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0174299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0174364Z res = mod(**inputs) 2025-11-03T16:40:29.0174613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0174679Z outputs = self.model( 2025-11-03T16:40:29.0174921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0175002Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0175218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0175302Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0175538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0175606Z return func(*args, **kwargs) 2025-11-03T16:40:29.0175874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0175993Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0176209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0176278Z return self.act(input) 2025-11-03T16:40:29.0176282Z 2025-11-03T16:40:29.0176390Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0176599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0176662Z res = mod(**inputs) 2025-11-03T16:40:29.0176917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0176983Z outputs = self.model( 2025-11-03T16:40:29.0177235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0177305Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0177524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0177608Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0177845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0177924Z return func(*args, **kwargs) 2025-11-03T16:40:29.0178166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0178253Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0178257Z 2025-11-03T16:40:29.0178356Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0178551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0178623Z res = mod(**inputs) 2025-11-03T16:40:29.0178864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0178960Z outputs = self.model( 2025-11-03T16:40:29.0179202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0179272Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0179495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0179572Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0179814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0179882Z return func(*args, **kwargs) 2025-11-03T16:40:29.0180136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0180244Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0180482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0180555Z return func(*args, **kwargs) 2025-11-03T16:40:29.0180798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0180919Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0180923Z 2025-11-03T16:40:29.0181026Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0181219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0181289Z res = mod(**inputs) 2025-11-03T16:40:29.0181557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0181635Z outputs = self.model( 2025-11-03T16:40:29.0181875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0181948Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0182169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0182246Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0182507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0182574Z return func(*args, **kwargs) 2025-11-03T16:40:29.0182815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0182921Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0183158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0183231Z return func(*args, **kwargs) 2025-11-03T16:40:29.0183472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0183561Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0183565Z 2025-11-03T16:40:29.0183666Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0183861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0183935Z res = mod(**inputs) 2025-11-03T16:40:29.0184175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0184248Z outputs = self.model( 2025-11-03T16:40:29.0184485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0184557Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0184781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0184876Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0185115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0185184Z return func(*args, **kwargs) 2025-11-03T16:40:29.0185433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0185528Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0185762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0185838Z return func(*args, **kwargs) 2025-11-03T16:40:29.0186097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0186216Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0186221Z 2025-11-03T16:40:29.0186325Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0186522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0186595Z res = mod(**inputs) 2025-11-03T16:40:29.0186840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0186919Z outputs = self.model( 2025-11-03T16:40:29.0187167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0187240Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0187489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0187568Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0187809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0187877Z return func(*args, **kwargs) 2025-11-03T16:40:29.0188122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0188217Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0188472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0188547Z return func(*args, **kwargs) 2025-11-03T16:40:29.0188798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0188942Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0188946Z 2025-11-03T16:40:29.0189047Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0189240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0189323Z res = mod(**inputs) 2025-11-03T16:40:29.0189555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0189625Z outputs = self.model( 2025-11-03T16:40:29.0189866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0189940Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0190147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0190221Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0190461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0190525Z return func(*args, **kwargs) 2025-11-03T16:40:29.0190781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0190873Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0191099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0191172Z return func(*args, **kwargs) 2025-11-03T16:40:29.0191404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0191494Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0191497Z 2025-11-03T16:40:29.0191592Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0191782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0191856Z res = mod(**inputs) 2025-11-03T16:40:29.0192084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0192156Z outputs = self.model( 2025-11-03T16:40:29.0192396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0192471Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0192680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0192755Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0192989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0193055Z return func(*args, **kwargs) 2025-11-03T16:40:29.0193310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0193404Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0193639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0193713Z return func(*args, **kwargs) 2025-11-03T16:40:29.0193959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0194079Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0194083Z 2025-11-03T16:40:29.0194266Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0194475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0194541Z res = mod(**inputs) 2025-11-03T16:40:29.0194792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0194870Z outputs = self.model( 2025-11-03T16:40:29.0195125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0195200Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0195405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0195479Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0195713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0195777Z return func(*args, **kwargs) 2025-11-03T16:40:29.0196011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0196102Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0196327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0196401Z return func(*args, **kwargs) 2025-11-03T16:40:29.0196674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0196802Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0196806Z 2025-11-03T16:40:29.0196900Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0197089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0197150Z res = mod(**inputs) 2025-11-03T16:40:29.0197379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0197451Z outputs = self.model( 2025-11-03T16:40:29.0197698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0197775Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0197979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0198053Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0198287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0198352Z return func(*args, **kwargs) 2025-11-03T16:40:29.0198588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0198678Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0198913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0198978Z return func(*args, **kwargs) 2025-11-03T16:40:29.0199219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0199307Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0199310Z 2025-11-03T16:40:29.0199404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0199596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0199656Z res = mod(**inputs) 2025-11-03T16:40:29.0199904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0199972Z outputs = self.model( 2025-11-03T16:40:29.0200200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0200274Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0200480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0200553Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0200784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0200846Z return func(*args, **kwargs) 2025-11-03T16:40:29.0201080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0201195Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0201198Z 2025-11-03T16:40:29.0201298Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0201481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0201543Z res = mod(**inputs) 2025-11-03T16:40:29.0201781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0201845Z outputs = self.model( 2025-11-03T16:40:29.0202077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0202160Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0202367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0202448Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0202675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0202747Z return func(*args, **kwargs) 2025-11-03T16:40:29.0202978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0203090Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0203327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0203393Z return self.act(input) 2025-11-03T16:40:29.0203398Z 2025-11-03T16:40:29.0203503Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0203687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0203754Z res = mod(**inputs) 2025-11-03T16:40:29.0203988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0204053Z outputs = self.model( 2025-11-03T16:40:29.0204288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0204354Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0204564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0204654Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0204883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0204957Z return func(*args, **kwargs) 2025-11-03T16:40:29.0205187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0205270Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0205288Z 2025-11-03T16:40:29.0205384Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0205575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0205636Z res = mod(**inputs) 2025-11-03T16:40:29.0205868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0205937Z outputs = self.model( 2025-11-03T16:40:29.0206170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0206247Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0206455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0206527Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0206763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0206830Z return func(*args, **kwargs) 2025-11-03T16:40:29.0207067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0207142Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0207145Z 2025-11-03T16:40:29.0207241Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0207436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0207496Z res = mod(**inputs) 2025-11-03T16:40:29.0207749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0207811Z outputs = self.model( 2025-11-03T16:40:29.0208051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0208116Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0208322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0208400Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0208634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0208706Z return func(*args, **kwargs) 2025-11-03T16:40:29.0208955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0209049Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0209281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0209345Z return func(*args, **kwargs) 2025-11-03T16:40:29.0209581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0209687Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0209691Z 2025-11-03T16:40:29.0209793Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0209976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0210035Z res = mod(**inputs) 2025-11-03T16:40:29.0210288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0210354Z outputs = self.model( 2025-11-03T16:40:29.0210592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0210658Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0210861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0210962Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0211191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0211264Z return func(*args, **kwargs) 2025-11-03T16:40:29.0211497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0211594Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0211829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0211897Z return func(*args, **kwargs) 2025-11-03T16:40:29.0212135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0212213Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0212216Z 2025-11-03T16:40:29.0212320Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0212506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0212567Z res = mod(**inputs) 2025-11-03T16:40:29.0212804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0212868Z outputs = self.model( 2025-11-03T16:40:29.0213108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0213177Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0213521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0213606Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0213832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0213909Z return func(*args, **kwargs) 2025-11-03T16:40:29.0214138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0214230Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0214471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0214537Z return func(*args, **kwargs) 2025-11-03T16:40:29.0214814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0214921Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0214925Z 2025-11-03T16:40:29.0215027Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0215214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0215276Z res = mod(**inputs) 2025-11-03T16:40:29.0215519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0215584Z outputs = self.model( 2025-11-03T16:40:29.0215824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0215892Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0216131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0216216Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0216447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0216523Z return func(*args, **kwargs) 2025-11-03T16:40:29.0216771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0216892Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0217151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0217216Z return func(*args, **kwargs) 2025-11-03T16:40:29.0217459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0217593Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0217596Z 2025-11-03T16:40:29.0217700Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0217890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0217952Z res = mod(**inputs) 2025-11-03T16:40:29.0218198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0218266Z outputs = self.model( 2025-11-03T16:40:29.0218508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0218578Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0218791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0218873Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0219107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0219206Z return func(*args, **kwargs) 2025-11-03T16:40:29.0219440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0219542Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0219775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0219843Z return func(*args, **kwargs) 2025-11-03T16:40:29.0220087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0220171Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0220174Z 2025-11-03T16:40:29.0220283Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0220492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0220553Z res = mod(**inputs) 2025-11-03T16:40:29.0220793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0220857Z outputs = self.model( 2025-11-03T16:40:29.0221096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0221164Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0221372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0221454Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0221681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0221753Z return func(*args, **kwargs) 2025-11-03T16:40:29.0222002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0222105Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0222334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0222399Z return func(*args, **kwargs) 2025-11-03T16:40:29.0222642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0222751Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0222754Z 2025-11-03T16:40:29.0222860Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0223048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0223110Z res = mod(**inputs) 2025-11-03T16:40:29.0223357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0223422Z outputs = self.model( 2025-11-03T16:40:29.0223666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0223735Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0223952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0224029Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0224259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0224333Z return func(*args, **kwargs) 2025-11-03T16:40:29.0224567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0224669Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0224899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0224993Z return func(*args, **kwargs) 2025-11-03T16:40:29.0225242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0225364Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0225367Z 2025-11-03T16:40:29.0225474Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0225662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0225730Z res = mod(**inputs) 2025-11-03T16:40:29.0225968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0226032Z outputs = self.model( 2025-11-03T16:40:29.0226297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0226367Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0226590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0226667Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0226899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0226975Z return func(*args, **kwargs) 2025-11-03T16:40:29.0227207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0227308Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0227538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0227619Z return func(*args, **kwargs) 2025-11-03T16:40:29.0227868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0227949Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0227952Z 2025-11-03T16:40:29.0228055Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0228245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0228332Z res = mod(**inputs) 2025-11-03T16:40:29.0228567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0228630Z outputs = self.model( 2025-11-03T16:40:29.0228875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0228944Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0229166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0229241Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0229473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0229548Z return func(*args, **kwargs) 2025-11-03T16:40:29.0229784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0229908Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0229911Z 2025-11-03T16:40:29.0230007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0230202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0230263Z res = mod(**inputs) 2025-11-03T16:40:29.0230506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0230582Z outputs = self.model( 2025-11-03T16:40:29.0230839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0230911Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0231129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0231211Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0231460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0231528Z return func(*args, **kwargs) 2025-11-03T16:40:29.0231773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0231891Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0232113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0232187Z return self.act(input) 2025-11-03T16:40:29.0232191Z 2025-11-03T16:40:29.0232288Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0232486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0232546Z res = mod(**inputs) 2025-11-03T16:40:29.0232799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0232863Z outputs = self.model( 2025-11-03T16:40:29.0233105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0233178Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0233407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0233496Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0233743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0233811Z return func(*args, **kwargs) 2025-11-03T16:40:29.0234068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0234202Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0234227Z 2025-11-03T16:40:29.0234341Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0234537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0234602Z res = mod(**inputs) 2025-11-03T16:40:29.0234859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0234932Z outputs = self.model( 2025-11-03T16:40:29.0235190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0235263Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0235493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0235571Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0235808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0235888Z return func(*args, **kwargs) 2025-11-03T16:40:29.0236132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0236239Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0236480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0236551Z return func(*args, **kwargs) 2025-11-03T16:40:29.0236801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0236931Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0236934Z 2025-11-03T16:40:29.0237048Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0237245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0237319Z res = mod(**inputs) 2025-11-03T16:40:29.0237566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0237631Z outputs = self.model( 2025-11-03T16:40:29.0237881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0237968Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0238193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0238274Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0238511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0238586Z return func(*args, **kwargs) 2025-11-03T16:40:29.0238827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0238932Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0239169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0239237Z return func(*args, **kwargs) 2025-11-03T16:40:29.0239504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0239585Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0239590Z 2025-11-03T16:40:29.0239696Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0239891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0239963Z res = mod(**inputs) 2025-11-03T16:40:29.0240211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0240293Z outputs = self.model( 2025-11-03T16:40:29.0240545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0240615Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0240843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0240921Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0241161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0241239Z return func(*args, **kwargs) 2025-11-03T16:40:29.0241483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0241586Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0241825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0241892Z return func(*args, **kwargs) 2025-11-03T16:40:29.0242141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0242249Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0242253Z 2025-11-03T16:40:29.0242362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0242555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0242643Z res = mod(**inputs) 2025-11-03T16:40:29.0242891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0242957Z outputs = self.model( 2025-11-03T16:40:29.0243210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0243283Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0243508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0243585Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0243823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0243913Z return func(*args, **kwargs) 2025-11-03T16:40:29.0244160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0244264Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0244506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0244588Z return func(*args, **kwargs) 2025-11-03T16:40:29.0244824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0244951Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0244954Z 2025-11-03T16:40:29.0245059Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0245264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0245336Z res = mod(**inputs) 2025-11-03T16:40:29.0245572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0245640Z outputs = self.model( 2025-11-03T16:40:29.0245889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0245957Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0246171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0246268Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0246493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0246564Z return func(*args, **kwargs) 2025-11-03T16:40:29.0246796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0246895Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0247121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0247193Z return func(*args, **kwargs) 2025-11-03T16:40:29.0247425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0247510Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0247513Z 2025-11-03T16:40:29.0247620Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0247809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0247879Z res = mod(**inputs) 2025-11-03T16:40:29.0248125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0248192Z outputs = self.model( 2025-11-03T16:40:29.0248436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0248524Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0248738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0248813Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0249048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0249114Z return func(*args, **kwargs) 2025-11-03T16:40:29.0249356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0249455Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0249697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0249770Z return func(*args, **kwargs) 2025-11-03T16:40:29.0250008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0250101Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0250105Z 2025-11-03T16:40:29.0250210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0250399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0250468Z res = mod(**inputs) 2025-11-03T16:40:29.0250705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0250770Z outputs = self.model( 2025-11-03T16:40:29.0251028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0251101Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0251318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0251395Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0251631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0251698Z return func(*args, **kwargs) 2025-11-03T16:40:29.0251949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0252051Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0252283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0252357Z return func(*args, **kwargs) 2025-11-03T16:40:29.0252592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0252712Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0252717Z 2025-11-03T16:40:29.0252823Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0253012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0253081Z res = mod(**inputs) 2025-11-03T16:40:29.0253317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0253390Z outputs = self.model( 2025-11-03T16:40:29.0253624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0253692Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0253912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0253988Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0254223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0254304Z return func(*args, **kwargs) 2025-11-03T16:40:29.0254537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0254637Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0254866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0254938Z return func(*args, **kwargs) 2025-11-03T16:40:29.0255171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0255248Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0255259Z 2025-11-03T16:40:29.0255371Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0255561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0255630Z res = mod(**inputs) 2025-11-03T16:40:29.0255865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0255937Z outputs = self.model( 2025-11-03T16:40:29.0256171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0256240Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0256460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0256533Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0256786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0256853Z return func(*args, **kwargs) 2025-11-03T16:40:29.0257092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0257221Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0257225Z 2025-11-03T16:40:29.0257325Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0257524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0257600Z res = mod(**inputs) 2025-11-03T16:40:29.0257838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0257911Z outputs = self.model( 2025-11-03T16:40:29.0258146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0258225Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0258437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0258523Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0258753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0258820Z return func(*args, **kwargs) 2025-11-03T16:40:29.0259063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0259178Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0259392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0259459Z return self.act(input) 2025-11-03T16:40:29.0259462Z 2025-11-03T16:40:29.0259561Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0259755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0259834Z res = mod(**inputs) 2025-11-03T16:40:29.0260077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0260141Z outputs = self.model( 2025-11-03T16:40:29.0260386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0260455Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0260665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0260749Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0260980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0261054Z return func(*args, **kwargs) 2025-11-03T16:40:29.0261303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0261384Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0261388Z 2025-11-03T16:40:29.0261492Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0261681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0261753Z res = mod(**inputs) 2025-11-03T16:40:29.0261991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0262056Z outputs = self.model( 2025-11-03T16:40:29.0262296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0262366Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0262603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0262679Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0262916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0262982Z return func(*args, **kwargs) 2025-11-03T16:40:29.0263221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0263329Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0263332Z 2025-11-03T16:40:29.0263430Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0263620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0263683Z res = mod(**inputs) 2025-11-03T16:40:29.0263918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0263990Z outputs = self.model( 2025-11-03T16:40:29.0264222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0264299Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0264507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0264581Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0264819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0264884Z return func(*args, **kwargs) 2025-11-03T16:40:29.0265124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0265216Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0265450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0265536Z return func(*args, **kwargs) 2025-11-03T16:40:29.0265773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0265886Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0265890Z 2025-11-03T16:40:29.0265988Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0266187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0266248Z res = mod(**inputs) 2025-11-03T16:40:29.0266490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0266560Z outputs = self.model( 2025-11-03T16:40:29.0266818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0266897Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0267107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0267189Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0267424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0267491Z return func(*args, **kwargs) 2025-11-03T16:40:29.0267731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0267823Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0268060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0268125Z return func(*args, **kwargs) 2025-11-03T16:40:29.0268385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0268472Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0268476Z 2025-11-03T16:40:29.0268571Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0268766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0268826Z res = mod(**inputs) 2025-11-03T16:40:29.0269074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0269144Z outputs = self.model( 2025-11-03T16:40:29.0269375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0269449Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0269661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0269743Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0269971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0270036Z return func(*args, **kwargs) 2025-11-03T16:40:29.0270275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0270370Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0270607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0270671Z return func(*args, **kwargs) 2025-11-03T16:40:29.0270907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0271024Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0271028Z 2025-11-03T16:40:29.0271128Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0271343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0271406Z res = mod(**inputs) 2025-11-03T16:40:29.0271655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0271721Z outputs = self.model( 2025-11-03T16:40:29.0271963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0272044Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0272261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0272347Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0272599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0272667Z return func(*args, **kwargs) 2025-11-03T16:40:29.0272920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0273016Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0273262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0273331Z return func(*args, **kwargs) 2025-11-03T16:40:29.0273573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0273712Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0273715Z 2025-11-03T16:40:29.0273816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0274037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0274166Z res = mod(**inputs) 2025-11-03T16:40:29.0274437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0274508Z outputs = self.model( 2025-11-03T16:40:29.0274759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0274866Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0275088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0275177Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0275425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0275495Z return func(*args, **kwargs) 2025-11-03T16:40:29.0275761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0275861Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0276124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0276192Z return func(*args, **kwargs) 2025-11-03T16:40:29.0276435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0276533Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0276537Z 2025-11-03T16:40:29.0276639Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0276842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0276906Z res = mod(**inputs) 2025-11-03T16:40:29.0277160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0277226Z outputs = self.model( 2025-11-03T16:40:29.0277491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0277568Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0277784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0277869Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0278106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0278171Z return func(*args, **kwargs) 2025-11-03T16:40:29.0278423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0278518Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0278779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0278848Z return func(*args, **kwargs) 2025-11-03T16:40:29.0279099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0279194Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0279197Z 2025-11-03T16:40:29.0279295Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0279500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0279563Z res = mod(**inputs) 2025-11-03T16:40:29.0279814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0279880Z outputs = self.model( 2025-11-03T16:40:29.0280139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0280218Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0280435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0280519Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0280757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0280838Z return func(*args, **kwargs) 2025-11-03T16:40:29.0281090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0281188Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0281437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0281507Z return func(*args, **kwargs) 2025-11-03T16:40:29.0281760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0281885Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0281889Z 2025-11-03T16:40:29.0281987Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0282188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0282252Z res = mod(**inputs) 2025-11-03T16:40:29.0282505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0282571Z outputs = self.model( 2025-11-03T16:40:29.0282823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0282910Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0283135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0283219Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0283472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0283545Z return func(*args, **kwargs) 2025-11-03T16:40:29.0283790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0283886Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0284131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0284199Z return func(*args, **kwargs) 2025-11-03T16:40:29.0284446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0284530Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0284549Z 2025-11-03T16:40:29.0284654Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0284863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0284931Z res = mod(**inputs) 2025-11-03T16:40:29.0285183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0285255Z outputs = self.model( 2025-11-03T16:40:29.0285500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0285581Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0285803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0285891Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0286145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0286221Z return func(*args, **kwargs) 2025-11-03T16:40:29.0286463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0286580Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0286583Z 2025-11-03T16:40:29.0286692Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0286914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0286986Z res = mod(**inputs) 2025-11-03T16:40:29.0287229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0287296Z outputs = self.model( 2025-11-03T16:40:29.0287548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0287621Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0287845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0287923Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0288171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0288238Z return func(*args, **kwargs) 2025-11-03T16:40:29.0288480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0288606Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0288818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0288895Z return self.act(input) 2025-11-03T16:40:29.0288901Z 2025-11-03T16:40:29.0289002Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0289194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0289288Z res = mod(**inputs) 2025-11-03T16:40:29.0289530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0289601Z outputs = self.model( 2025-11-03T16:40:29.0289853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0289923Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0290136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0290210Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0290442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0290533Z return func(*args, **kwargs) 2025-11-03T16:40:29.0290772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0290851Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0290855Z 2025-11-03T16:40:29.0290950Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0291140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0291200Z res = mod(**inputs) 2025-11-03T16:40:29.0291436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0291498Z outputs = self.model( 2025-11-03T16:40:29.0291724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0291811Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0292016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0292099Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0292322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0292386Z return func(*args, **kwargs) 2025-11-03T16:40:29.0292623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0292730Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0292962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0293028Z return func(*args, **kwargs) 2025-11-03T16:40:29.0293266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0293371Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0293376Z 2025-11-03T16:40:29.0293472Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0293663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0293723Z res = mod(**inputs) 2025-11-03T16:40:29.0293959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0294023Z outputs = self.model( 2025-11-03T16:40:29.0294249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0294323Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0294528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0294615Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0294837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0294924Z return func(*args, **kwargs) 2025-11-03T16:40:29.0295157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0295249Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0295486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0295549Z return func(*args, **kwargs) 2025-11-03T16:40:29.0295789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0295864Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0295867Z 2025-11-03T16:40:29.0295964Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0296170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0296232Z res = mod(**inputs) 2025-11-03T16:40:29.0296468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0296530Z outputs = self.model( 2025-11-03T16:40:29.0296758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0296832Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0297036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0297116Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0297340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0297432Z return func(*args, **kwargs) 2025-11-03T16:40:29.0297659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0297750Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0297982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0298047Z return func(*args, **kwargs) 2025-11-03T16:40:29.0298299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0298401Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0298404Z 2025-11-03T16:40:29.0298500Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0298690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0298751Z res = mod(**inputs) 2025-11-03T16:40:29.0298987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0299052Z outputs = self.model( 2025-11-03T16:40:29.0299287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0299355Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0299562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0299645Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0299866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0299938Z return func(*args, **kwargs) 2025-11-03T16:40:29.0300169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0300261Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0300494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0300571Z return func(*args, **kwargs) 2025-11-03T16:40:29.0300808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0300933Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0300938Z 2025-11-03T16:40:29.0301032Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0301222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0301282Z res = mod(**inputs) 2025-11-03T16:40:29.0301530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0301594Z outputs = self.model( 2025-11-03T16:40:29.0301847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0301918Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0302123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0302203Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0302425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0302498Z return func(*args, **kwargs) 2025-11-03T16:40:29.0302728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0302819Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0303066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0303131Z return func(*args, **kwargs) 2025-11-03T16:40:29.0303370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0303453Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0303457Z 2025-11-03T16:40:29.0303561Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0303741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0303819Z res = mod(**inputs) 2025-11-03T16:40:29.0304058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0304121Z outputs = self.model( 2025-11-03T16:40:29.0304361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0304429Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0304644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0304725Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0304953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0305025Z return func(*args, **kwargs) 2025-11-03T16:40:29.0305256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0305346Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0305580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0305645Z return func(*args, **kwargs) 2025-11-03T16:40:29.0305885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0305975Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0305994Z 2025-11-03T16:40:29.0306098Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0306282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0306341Z res = mod(**inputs) 2025-11-03T16:40:29.0306578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0306643Z outputs = self.model( 2025-11-03T16:40:29.0306879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0306946Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0307153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0307248Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0307474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0307547Z return func(*args, **kwargs) 2025-11-03T16:40:29.0307777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0307868Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0308098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0308161Z return func(*args, **kwargs) 2025-11-03T16:40:29.0308395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0308512Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0308515Z 2025-11-03T16:40:29.0308633Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0308817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0308877Z res = mod(**inputs) 2025-11-03T16:40:29.0309113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0309174Z outputs = self.model( 2025-11-03T16:40:29.0309410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0309492Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0309698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0309778Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0310003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0310075Z return func(*args, **kwargs) 2025-11-03T16:40:29.0310305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0310404Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0310632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0310696Z return func(*args, **kwargs) 2025-11-03T16:40:29.0310934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0311011Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0311014Z 2025-11-03T16:40:29.0311114Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0311298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0311360Z res = mod(**inputs) 2025-11-03T16:40:29.0311609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0311698Z outputs = self.model( 2025-11-03T16:40:29.0311945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0312015Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0312229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0312312Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0312547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0312622Z return func(*args, **kwargs) 2025-11-03T16:40:29.0312873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0313010Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0313016Z 2025-11-03T16:40:29.0313115Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0313521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0313597Z res = mod(**inputs) 2025-11-03T16:40:29.0313841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0313919Z outputs = self.model( 2025-11-03T16:40:29.0314205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0314292Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0314514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0314630Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0314875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0314944Z return func(*args, **kwargs) 2025-11-03T16:40:29.0315226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0315352Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0315580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0315656Z return self.act(input) 2025-11-03T16:40:29.0315660Z 2025-11-03T16:40:29.0315758Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0315960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0316023Z res = mod(**inputs) 2025-11-03T16:40:29.0316275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0316352Z outputs = self.model( 2025-11-03T16:40:29.0316590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0316668Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0316883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0316960Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0317203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0317269Z return func(*args, **kwargs) 2025-11-03T16:40:29.0317515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0317597Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0317602Z 2025-11-03T16:40:29.0317705Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0317916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0317977Z res = mod(**inputs) 2025-11-03T16:40:29.0318220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0318283Z outputs = self.model( 2025-11-03T16:40:29.0318529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0318598Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0318810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0318892Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0319150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0319224Z return func(*args, **kwargs) 2025-11-03T16:40:29.0319464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0319540Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0319550Z 2025-11-03T16:40:29.0319646Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0319836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0319904Z res = mod(**inputs) 2025-11-03T16:40:29.0320140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0320211Z outputs = self.model( 2025-11-03T16:40:29.0320463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0320535Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0320752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0320828Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0321065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0321131Z return func(*args, **kwargs) 2025-11-03T16:40:29.0321385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0321486Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0321716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0321788Z return func(*args, **kwargs) 2025-11-03T16:40:29.0322027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0322132Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0322144Z 2025-11-03T16:40:29.0322241Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0322426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0322494Z res = mod(**inputs) 2025-11-03T16:40:29.0322733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0322802Z outputs = self.model( 2025-11-03T16:40:29.0323040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0323110Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0323325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0323401Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0323640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0323721Z return func(*args, **kwargs) 2025-11-03T16:40:29.0323966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0324067Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0324309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0324382Z return func(*args, **kwargs) 2025-11-03T16:40:29.0324622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0324706Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0324710Z 2025-11-03T16:40:29.0324823Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0325011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0325080Z res = mod(**inputs) 2025-11-03T16:40:29.0325328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0325397Z outputs = self.model( 2025-11-03T16:40:29.0325625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0325693Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0325905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0325978Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0326224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0326289Z return func(*args, **kwargs) 2025-11-03T16:40:29.0326521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0326619Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0326843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0326930Z return func(*args, **kwargs) 2025-11-03T16:40:29.0327159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0327267Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0327271Z 2025-11-03T16:40:29.0327365Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0327552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0327622Z res = mod(**inputs) 2025-11-03T16:40:29.0327854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0327926Z outputs = self.model( 2025-11-03T16:40:29.0328154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0328222Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0328434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0328507Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0328737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0328801Z return func(*args, **kwargs) 2025-11-03T16:40:29.0329039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0329130Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0329373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0329445Z return func(*args, **kwargs) 2025-11-03T16:40:29.0329676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0329811Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0329814Z 2025-11-03T16:40:29.0329910Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0330095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0330161Z res = mod(**inputs) 2025-11-03T16:40:29.0330411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0330481Z outputs = self.model( 2025-11-03T16:40:29.0330712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0330780Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0330994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0331067Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0331302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0331366Z return func(*args, **kwargs) 2025-11-03T16:40:29.0331602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0331691Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0331930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0332005Z return func(*args, **kwargs) 2025-11-03T16:40:29.0332234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0332320Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0332324Z 2025-11-03T16:40:29.0332419Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0332627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0332692Z res = mod(**inputs) 2025-11-03T16:40:29.0332920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0332990Z outputs = self.model( 2025-11-03T16:40:29.0333220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0333295Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0333499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0333573Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0333802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0333864Z return func(*args, **kwargs) 2025-11-03T16:40:29.0334102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0334190Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0334412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0334484Z return func(*args, **kwargs) 2025-11-03T16:40:29.0334712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0334825Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0334829Z 2025-11-03T16:40:29.0334924Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0335108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0335174Z res = mod(**inputs) 2025-11-03T16:40:29.0335405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0335477Z outputs = self.model( 2025-11-03T16:40:29.0335705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0335777Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0336000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0336074Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0336305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0336369Z return func(*args, **kwargs) 2025-11-03T16:40:29.0336602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0336694Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0336917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0336987Z return func(*args, **kwargs) 2025-11-03T16:40:29.0337214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0337367Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0337372Z 2025-11-03T16:40:29.0337468Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0337659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0337718Z res = mod(**inputs) 2025-11-03T16:40:29.0337947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0338016Z outputs = self.model( 2025-11-03T16:40:29.0338258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0338333Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0338538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0338612Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0338844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0338907Z return func(*args, **kwargs) 2025-11-03T16:40:29.0339142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0339233Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0339458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0339533Z return func(*args, **kwargs) 2025-11-03T16:40:29.0339763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0339847Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0339851Z 2025-11-03T16:40:29.0339948Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0340142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0340203Z res = mod(**inputs) 2025-11-03T16:40:29.0340439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0340529Z outputs = self.model( 2025-11-03T16:40:29.0340762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0340837Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0341049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0341121Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0341357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0341421Z return func(*args, **kwargs) 2025-11-03T16:40:29.0341679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0341793Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0341798Z 2025-11-03T16:40:29.0341901Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0342085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0342145Z res = mod(**inputs) 2025-11-03T16:40:29.0342381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0342445Z outputs = self.model( 2025-11-03T16:40:29.0342680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0342747Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0342970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0343055Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0343279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0343351Z return func(*args, **kwargs) 2025-11-03T16:40:29.0343581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0343693Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0343920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0343991Z return self.act(input) 2025-11-03T16:40:29.0343995Z 2025-11-03T16:40:29.0344099Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0344284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0344353Z res = mod(**inputs) 2025-11-03T16:40:29.0344587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0344651Z outputs = self.model( 2025-11-03T16:40:29.0344888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0344954Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0345167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0345242Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0345468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0345540Z return func(*args, **kwargs) 2025-11-03T16:40:29.0345770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0345857Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0345860Z 2025-11-03T16:40:29.0345973Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0346159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0346224Z res = mod(**inputs) 2025-11-03T16:40:29.0346454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0346527Z outputs = self.model( 2025-11-03T16:40:29.0346764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0346838Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0347047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0347122Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0347366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0347432Z return func(*args, **kwargs) 2025-11-03T16:40:29.0347669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0347760Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0347985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0348059Z return func(*args, **kwargs) 2025-11-03T16:40:29.0348287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0348397Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0348400Z 2025-11-03T16:40:29.0348508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0348700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0348760Z res = mod(**inputs) 2025-11-03T16:40:29.0348989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0349058Z outputs = self.model( 2025-11-03T16:40:29.0349287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0349374Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0349585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0349658Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0349895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0349959Z return func(*args, **kwargs) 2025-11-03T16:40:29.0350200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0350293Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0350519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0350586Z return func(*args, **kwargs) 2025-11-03T16:40:29.0350820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0350902Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0350905Z 2025-11-03T16:40:29.0351001Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0351192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0351250Z res = mod(**inputs) 2025-11-03T16:40:29.0351492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0351577Z outputs = self.model( 2025-11-03T16:40:29.0351807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0351880Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0352089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0352164Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0352403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0352467Z return func(*args, **kwargs) 2025-11-03T16:40:29.0352701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0352810Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0353033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0353106Z return func(*args, **kwargs) 2025-11-03T16:40:29.0353333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0353441Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0353446Z 2025-11-03T16:40:29.0353541Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0353728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0353787Z res = mod(**inputs) 2025-11-03T16:40:29.0354028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0354189Z outputs = self.model( 2025-11-03T16:40:29.0354447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0354529Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0354744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0354821Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0355103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0355186Z return func(*args, **kwargs) 2025-11-03T16:40:29.0355430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0355525Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0355764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0355833Z return func(*args, **kwargs) 2025-11-03T16:40:29.0356069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0356209Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0356213Z 2025-11-03T16:40:29.0356312Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0356508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0356573Z res = mod(**inputs) 2025-11-03T16:40:29.0356811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0356883Z outputs = self.model( 2025-11-03T16:40:29.0357119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0357197Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0357409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0357510Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0357748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0357813Z return func(*args, **kwargs) 2025-11-03T16:40:29.0358056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0358148Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0358387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0358451Z return func(*args, **kwargs) 2025-11-03T16:40:29.0358702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0358795Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0358800Z 2025-11-03T16:40:29.0358897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0359092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0359152Z res = mod(**inputs) 2025-11-03T16:40:29.0359391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0359463Z outputs = self.model( 2025-11-03T16:40:29.0359703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0359779Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0359992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0360089Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0360319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0360386Z return func(*args, **kwargs) 2025-11-03T16:40:29.0360628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0360720Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0360972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0361036Z return func(*args, **kwargs) 2025-11-03T16:40:29.0361272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0361373Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0361376Z 2025-11-03T16:40:29.0361473Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0361671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0361733Z res = mod(**inputs) 2025-11-03T16:40:29.0361970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0362043Z outputs = self.model( 2025-11-03T16:40:29.0362281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0362358Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0362568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0362650Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0362878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0362945Z return func(*args, **kwargs) 2025-11-03T16:40:29.0363187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0363297Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0363537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0363601Z return func(*args, **kwargs) 2025-11-03T16:40:29.0363842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0363970Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0363973Z 2025-11-03T16:40:29.0364069Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0364265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0364328Z res = mod(**inputs) 2025-11-03T16:40:29.0364589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0364657Z outputs = self.model( 2025-11-03T16:40:29.0364892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0364966Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0365180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0365264Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0365495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0365562Z return func(*args, **kwargs) 2025-11-03T16:40:29.0365821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0365916Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0366148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0366215Z return func(*args, **kwargs) 2025-11-03T16:40:29.0366446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0366535Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0366555Z 2025-11-03T16:40:29.0366657Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0366859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0366925Z res = mod(**inputs) 2025-11-03T16:40:29.0367173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0367242Z outputs = self.model( 2025-11-03T16:40:29.0367482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0367563Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0367780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0367864Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0368098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0368169Z return func(*args, **kwargs) 2025-11-03T16:40:29.0368411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0368531Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0368534Z 2025-11-03T16:40:29.0368644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0368834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0368918Z res = mod(**inputs) 2025-11-03T16:40:29.0369154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0369220Z outputs = self.model( 2025-11-03T16:40:29.0369466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0369537Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0369752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0369825Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0370054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0370129Z return func(*args, **kwargs) 2025-11-03T16:40:29.0370377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0370501Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0370703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0370769Z return self.act(input) 2025-11-03T16:40:29.0370780Z 2025-11-03T16:40:29.0370879Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0371063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0371135Z res = mod(**inputs) 2025-11-03T16:40:29.0371367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0371438Z outputs = self.model( 2025-11-03T16:40:29.0371695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0371766Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0371984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0372059Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0372296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0372379Z return func(*args, **kwargs) 2025-11-03T16:40:29.0372614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0372703Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0372706Z 2025-11-03T16:40:29.0372802Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0372999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0373061Z res = mod(**inputs) 2025-11-03T16:40:29.0373295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0373368Z outputs = self.model( 2025-11-03T16:40:29.0373599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0373675Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0373888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0373972Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0374201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0374267Z return func(*args, **kwargs) 2025-11-03T16:40:29.0374510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0374587Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0374607Z 2025-11-03T16:40:29.0374714Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0374908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0374968Z res = mod(**inputs) 2025-11-03T16:40:29.0375216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0375284Z outputs = self.model( 2025-11-03T16:40:29.0375534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0375604Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0375823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0375930Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0376163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0376239Z return func(*args, **kwargs) 2025-11-03T16:40:29.0376519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0376617Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0376848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0376912Z return func(*args, **kwargs) 2025-11-03T16:40:29.0377159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0377265Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0377305Z 2025-11-03T16:40:29.0377410Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0377597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0377658Z res = mod(**inputs) 2025-11-03T16:40:29.0377912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0377980Z outputs = self.model( 2025-11-03T16:40:29.0378228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0378314Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0378540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0378617Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0378859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0378935Z return func(*args, **kwargs) 2025-11-03T16:40:29.0379178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0379283Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0379523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0379590Z return func(*args, **kwargs) 2025-11-03T16:40:29.0379843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0379921Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0379924Z 2025-11-03T16:40:29.0380031Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0380226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0380290Z res = mod(**inputs) 2025-11-03T16:40:29.0380542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0380625Z outputs = self.model( 2025-11-03T16:40:29.0380878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0380947Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0381172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0381250Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0381489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0381564Z return func(*args, **kwargs) 2025-11-03T16:40:29.0381818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0381938Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0382176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0382242Z return func(*args, **kwargs) 2025-11-03T16:40:29.0382491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0382600Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0382604Z 2025-11-03T16:40:29.0382711Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0382906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0382976Z res = mod(**inputs) 2025-11-03T16:40:29.0383245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0383315Z outputs = self.model( 2025-11-03T16:40:29.0383561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0383634Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0383858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0383935Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0384187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0384262Z return func(*args, **kwargs) 2025-11-03T16:40:29.0384509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0384611Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0384851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0384919Z return func(*args, **kwargs) 2025-11-03T16:40:29.0385172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0385305Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0385308Z 2025-11-03T16:40:29.0385416Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0385609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0385678Z res = mod(**inputs) 2025-11-03T16:40:29.0385919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0385985Z outputs = self.model( 2025-11-03T16:40:29.0386238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0386312Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0386540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0386632Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0386864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0386940Z return func(*args, **kwargs) 2025-11-03T16:40:29.0387178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0387280Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0387514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0387587Z return func(*args, **kwargs) 2025-11-03T16:40:29.0387844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0387930Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0387935Z 2025-11-03T16:40:29.0388043Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0388233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0388300Z res = mod(**inputs) 2025-11-03T16:40:29.0388541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0388606Z outputs = self.model( 2025-11-03T16:40:29.0388854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0388924Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0389162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0389242Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0389478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0389553Z return func(*args, **kwargs) 2025-11-03T16:40:29.0389800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0389897Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0390134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0390203Z return func(*args, **kwargs) 2025-11-03T16:40:29.0390429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0390518Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0390523Z 2025-11-03T16:40:29.0390628Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0390810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0390877Z res = mod(**inputs) 2025-11-03T16:40:29.0391103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0391165Z outputs = self.model( 2025-11-03T16:40:29.0391401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0391469Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0391680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0391752Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0391974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0392044Z return func(*args, **kwargs) 2025-11-03T16:40:29.0392272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0392391Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0392614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0392686Z return func(*args, **kwargs) 2025-11-03T16:40:29.0392917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0393035Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0393038Z 2025-11-03T16:40:29.0393140Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0393325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0393416Z res = mod(**inputs) 2025-11-03T16:40:29.0393647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0393712Z outputs = self.model( 2025-11-03T16:40:29.0393962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0394029Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0394323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0394407Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0394648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0394715Z return func(*args, **kwargs) 2025-11-03T16:40:29.0394971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0395076Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0395315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0395401Z return func(*args, **kwargs) 2025-11-03T16:40:29.0395635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0395732Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0395735Z 2025-11-03T16:40:29.0395839Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0396022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0396091Z res = mod(**inputs) 2025-11-03T16:40:29.0396320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0396384Z outputs = self.model( 2025-11-03T16:40:29.0396623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0396692Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0396902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0396976Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0397208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0397273Z return func(*args, **kwargs) 2025-11-03T16:40:29.0397499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0397618Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0397624Z 2025-11-03T16:40:29.0397720Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0397908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0397994Z res = mod(**inputs) 2025-11-03T16:40:29.0398221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0398291Z outputs = self.model( 2025-11-03T16:40:29.0398519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0398594Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0398798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0398883Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0399108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0399189Z return func(*args, **kwargs) 2025-11-03T16:40:29.0399427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0399539Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0399745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0399810Z return self.act(input) 2025-11-03T16:40:29.0399815Z 2025-11-03T16:40:29.0399910Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0400097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0400156Z res = mod(**inputs) 2025-11-03T16:40:29.0400393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0400469Z outputs = self.model( 2025-11-03T16:40:29.0400703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0400779Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0400986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0401066Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0401292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0401379Z return func(*args, **kwargs) 2025-11-03T16:40:29.0401617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0401693Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0401697Z 2025-11-03T16:40:29.0401801Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0401988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0402053Z res = mod(**inputs) 2025-11-03T16:40:29.0402287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0402351Z outputs = self.model( 2025-11-03T16:40:29.0402589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0402657Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0402870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0402943Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0403173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0403246Z return func(*args, **kwargs) 2025-11-03T16:40:29.0403479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0403595Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0403824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0403893Z return func(*args, **kwargs) 2025-11-03T16:40:29.0404126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0404230Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0404234Z 2025-11-03T16:40:29.0404335Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0404519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0404582Z res = mod(**inputs) 2025-11-03T16:40:29.0404834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0404898Z outputs = self.model( 2025-11-03T16:40:29.0405135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0405201Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0405426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0405500Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0405731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0405795Z return func(*args, **kwargs) 2025-11-03T16:40:29.0406026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0406140Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0406362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0406433Z return func(*args, **kwargs) 2025-11-03T16:40:29.0406660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0406735Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0406738Z 2025-11-03T16:40:29.0406855Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0407037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0407103Z res = mod(**inputs) 2025-11-03T16:40:29.0407330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0407392Z outputs = self.model( 2025-11-03T16:40:29.0407629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0407699Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0407910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0407983Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0408213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0408277Z return func(*args, **kwargs) 2025-11-03T16:40:29.0408503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0408602Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0408825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0408898Z return func(*args, **kwargs) 2025-11-03T16:40:29.0409126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0409245Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0409248Z 2025-11-03T16:40:29.0409349Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0409531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0409599Z res = mod(**inputs) 2025-11-03T16:40:29.0409829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0409898Z outputs = self.model( 2025-11-03T16:40:29.0410125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0410193Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0410422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0410498Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0410729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0410794Z return func(*args, **kwargs) 2025-11-03T16:40:29.0411024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0411126Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0411350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0411421Z return func(*args, **kwargs) 2025-11-03T16:40:29.0411664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0411794Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0411804Z 2025-11-03T16:40:29.0411902Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0412084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0412152Z res = mod(**inputs) 2025-11-03T16:40:29.0412389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0412475Z outputs = self.model( 2025-11-03T16:40:29.0412705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0412773Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0412988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0413065Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0413444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0413515Z return func(*args, **kwargs) 2025-11-03T16:40:29.0413789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0413894Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0414126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0414201Z return func(*args, **kwargs) 2025-11-03T16:40:29.0414436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0414529Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0414533Z 2025-11-03T16:40:29.0414632Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0414822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0414930Z res = mod(**inputs) 2025-11-03T16:40:29.0415164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0415236Z outputs = self.model( 2025-11-03T16:40:29.0415471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0415541Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0415770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0415844Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0416075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0416143Z return func(*args, **kwargs) 2025-11-03T16:40:29.0416388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0416489Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0416721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0416795Z return func(*args, **kwargs) 2025-11-03T16:40:29.0417033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0417134Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0417138Z 2025-11-03T16:40:29.0417236Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0417424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0417492Z res = mod(**inputs) 2025-11-03T16:40:29.0417747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0417820Z outputs = self.model( 2025-11-03T16:40:29.0418058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0418126Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0418346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0418453Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0418689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0418754Z return func(*args, **kwargs) 2025-11-03T16:40:29.0418990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0419095Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0419325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0419398Z return func(*args, **kwargs) 2025-11-03T16:40:29.0419634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0419763Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0419768Z 2025-11-03T16:40:29.0419864Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0420053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0420122Z res = mod(**inputs) 2025-11-03T16:40:29.0420358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0420431Z outputs = self.model( 2025-11-03T16:40:29.0420671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0420756Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0420974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0421050Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0421293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0421363Z return func(*args, **kwargs) 2025-11-03T16:40:29.0421611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0421705Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0421940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0422029Z return func(*args, **kwargs) 2025-11-03T16:40:29.0422273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0422361Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0422364Z 2025-11-03T16:40:29.0422464Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0422652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0422723Z res = mod(**inputs) 2025-11-03T16:40:29.0422964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0423036Z outputs = self.model( 2025-11-03T16:40:29.0423276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0423359Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0423582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0423659Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0423896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0423962Z return func(*args, **kwargs) 2025-11-03T16:40:29.0424203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0424333Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0424337Z 2025-11-03T16:40:29.0424434Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0424628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0424690Z res = mod(**inputs) 2025-11-03T16:40:29.0424937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0425001Z outputs = self.model( 2025-11-03T16:40:29.0425269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0425345Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0425552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0425636Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0425865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0425939Z return func(*args, **kwargs) 2025-11-03T16:40:29.0426173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0426288Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0426500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0426582Z return self.act(input) 2025-11-03T16:40:29.0426587Z 2025-11-03T16:40:29.0426688Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0426872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0426932Z res = mod(**inputs) 2025-11-03T16:40:29.0427190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0427255Z outputs = self.model( 2025-11-03T16:40:29.0427503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0427570Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0427797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0427881Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0428115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0428187Z return func(*args, **kwargs) 2025-11-03T16:40:29.0428426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0428513Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0428517Z 2025-11-03T16:40:29.0428623Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0428806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0428872Z res = mod(**inputs) 2025-11-03T16:40:29.0429150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0429224Z outputs = self.model( 2025-11-03T16:40:29.0429462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0429532Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0429753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0429828Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0430080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0430150Z return func(*args, **kwargs) 2025-11-03T16:40:29.0430387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0430476Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0430479Z 2025-11-03T16:40:29.0430582Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0430781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0430845Z res = mod(**inputs) 2025-11-03T16:40:29.0431097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0431163Z outputs = self.model( 2025-11-03T16:40:29.0431411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0431491Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0431701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0431786Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0432022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0432091Z return func(*args, **kwargs) 2025-11-03T16:40:29.0432338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0432448Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0432684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0432748Z return func(*args, **kwargs) 2025-11-03T16:40:29.0432985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0433097Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0433101Z 2025-11-03T16:40:29.0433196Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0433391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0433452Z res = mod(**inputs) 2025-11-03T16:40:29.0433711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0433778Z outputs = self.model( 2025-11-03T16:40:29.0434015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0434091Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0434396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0434489Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0434728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0434795Z return func(*args, **kwargs) 2025-11-03T16:40:29.0435073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0435170Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0435403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0435469Z return func(*args, **kwargs) 2025-11-03T16:40:29.0435706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0435784Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0435803Z 2025-11-03T16:40:29.0435899Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0436089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0436148Z res = mod(**inputs) 2025-11-03T16:40:29.0436388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0436453Z outputs = self.model( 2025-11-03T16:40:29.0436684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0436759Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0436964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0437043Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0437275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0437340Z return func(*args, **kwargs) 2025-11-03T16:40:29.0437580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0437672Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0437907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0437974Z return func(*args, **kwargs) 2025-11-03T16:40:29.0438209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0438327Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0438330Z 2025-11-03T16:40:29.0438425Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0438617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0438677Z res = mod(**inputs) 2025-11-03T16:40:29.0438909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0438972Z outputs = self.model( 2025-11-03T16:40:29.0439198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0439288Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0439493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0439576Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0439801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0439872Z return func(*args, **kwargs) 2025-11-03T16:40:29.0440114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0440205Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0440436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0440498Z return func(*args, **kwargs) 2025-11-03T16:40:29.0440762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0440889Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0440894Z 2025-11-03T16:40:29.0440988Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0441181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0441240Z res = mod(**inputs) 2025-11-03T16:40:29.0441480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0441556Z outputs = self.model( 2025-11-03T16:40:29.0441784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0441860Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0442074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0442157Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0442381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0442452Z return func(*args, **kwargs) 2025-11-03T16:40:29.0442682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0442776Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0443006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0443070Z return func(*args, **kwargs) 2025-11-03T16:40:29.0443309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0443390Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0443393Z 2025-11-03T16:40:29.0443489Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0443679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0443753Z res = mod(**inputs) 2025-11-03T16:40:29.0443995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0444057Z outputs = self.model( 2025-11-03T16:40:29.0444297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0444366Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0444574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0444652Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0444879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0444961Z return func(*args, **kwargs) 2025-11-03T16:40:29.0445188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0445281Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0445513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0445575Z return func(*args, **kwargs) 2025-11-03T16:40:29.0445811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0445899Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0445902Z 2025-11-03T16:40:29.0446000Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0446201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0446261Z res = mod(**inputs) 2025-11-03T16:40:29.0446505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0446568Z outputs = self.model( 2025-11-03T16:40:29.0446815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0446886Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0447094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0447190Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0447413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0447484Z return func(*args, **kwargs) 2025-11-03T16:40:29.0447714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0447807Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0448039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0448102Z return func(*args, **kwargs) 2025-11-03T16:40:29.0448337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0448456Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0448460Z 2025-11-03T16:40:29.0448563Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0448746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0448805Z res = mod(**inputs) 2025-11-03T16:40:29.0449044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0449108Z outputs = self.model( 2025-11-03T16:40:29.0449348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0449431Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0449635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0449717Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0449946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0450023Z return func(*args, **kwargs) 2025-11-03T16:40:29.0450259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0450353Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0450603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0450669Z return func(*args, **kwargs) 2025-11-03T16:40:29.0450908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0450986Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0450989Z 2025-11-03T16:40:29.0451092Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0451276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0451336Z res = mod(**inputs) 2025-11-03T16:40:29.0451574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0451637Z outputs = self.model( 2025-11-03T16:40:29.0451886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0451957Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0452162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0452246Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0452473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0452545Z return func(*args, **kwargs) 2025-11-03T16:40:29.0452791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0452902Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0452914Z 2025-11-03T16:40:29.0453007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0453189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0453265Z res = mod(**inputs) 2025-11-03T16:40:29.0453495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0453565Z outputs = self.model( 2025-11-03T16:40:29.0453796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0453861Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0454074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0454148Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0454377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0454440Z return func(*args, **kwargs) 2025-11-03T16:40:29.0454669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0454789Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0454987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0455079Z return self.act(input) 2025-11-03T16:40:29.0455083Z 2025-11-03T16:40:29.0455176Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0455365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0455426Z res = mod(**inputs) 2025-11-03T16:40:29.0455655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0455724Z outputs = self.model( 2025-11-03T16:40:29.0455953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0456027Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0456248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0456323Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0456557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0456621Z return func(*args, **kwargs) 2025-11-03T16:40:29.0456858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0456937Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0456940Z 2025-11-03T16:40:29.0457043Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0457226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0457286Z res = mod(**inputs) 2025-11-03T16:40:29.0457536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0457599Z outputs = self.model( 2025-11-03T16:40:29.0457835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0457901Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0458104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0458198Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0458420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0458492Z return func(*args, **kwargs) 2025-11-03T16:40:29.0458718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0458811Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0459039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0459104Z return func(*args, **kwargs) 2025-11-03T16:40:29.0459336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0459438Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0459442Z 2025-11-03T16:40:29.0459545Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0459725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0459783Z res = mod(**inputs) 2025-11-03T16:40:29.0460013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0460075Z outputs = self.model( 2025-11-03T16:40:29.0460312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0460376Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0460595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0460675Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0460896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0460970Z return func(*args, **kwargs) 2025-11-03T16:40:29.0461197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0461289Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0461518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0461583Z return func(*args, **kwargs) 2025-11-03T16:40:29.0461846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0461925Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0461928Z 2025-11-03T16:40:29.0462028Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0462212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0462272Z res = mod(**inputs) 2025-11-03T16:40:29.0462508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0462571Z outputs = self.model( 2025-11-03T16:40:29.0462806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0462873Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0463101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0463182Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0463405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0463475Z return func(*args, **kwargs) 2025-11-03T16:40:29.0463705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0463811Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0464041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0464106Z return func(*args, **kwargs) 2025-11-03T16:40:29.0464339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0464446Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0464449Z 2025-11-03T16:40:29.0464550Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0464734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0464792Z res = mod(**inputs) 2025-11-03T16:40:29.0465032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0465096Z outputs = self.model( 2025-11-03T16:40:29.0465331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0465397Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0465602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0465683Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0465911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0465996Z return func(*args, **kwargs) 2025-11-03T16:40:29.0466225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0466325Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0466548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0466613Z return func(*args, **kwargs) 2025-11-03T16:40:29.0466851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0466975Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0466978Z 2025-11-03T16:40:29.0467079Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0467301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0467362Z res = mod(**inputs) 2025-11-03T16:40:29.0467605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0467666Z outputs = self.model( 2025-11-03T16:40:29.0467912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0467982Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0468193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0468276Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0468505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0468593Z return func(*args, **kwargs) 2025-11-03T16:40:29.0468834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0468935Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0469167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0469232Z return func(*args, **kwargs) 2025-11-03T16:40:29.0469473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0469569Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0469573Z 2025-11-03T16:40:29.0469677Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0469864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0469924Z res = mod(**inputs) 2025-11-03T16:40:29.0470172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0470236Z outputs = self.model( 2025-11-03T16:40:29.0470483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0470550Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0470771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0470848Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0471085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0471160Z return func(*args, **kwargs) 2025-11-03T16:40:29.0471396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0471498Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0471731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0471811Z return func(*args, **kwargs) 2025-11-03T16:40:29.0472059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0472151Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0472154Z 2025-11-03T16:40:29.0472258Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0472445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0472506Z res = mod(**inputs) 2025-11-03T16:40:29.0472753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0472817Z outputs = self.model( 2025-11-03T16:40:29.0473079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0473149Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0473369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0473444Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0473675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0473749Z return func(*args, **kwargs) 2025-11-03T16:40:29.0473988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0474086Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0474389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0474480Z return func(*args, **kwargs) 2025-11-03T16:40:29.0474725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0474849Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0474853Z 2025-11-03T16:40:29.0474960Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0475153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0475259Z res = mod(**inputs) 2025-11-03T16:40:29.0475505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0475571Z outputs = self.model( 2025-11-03T16:40:29.0475817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0475889Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0476121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0476198Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0476428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0476502Z return func(*args, **kwargs) 2025-11-03T16:40:29.0476736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0476836Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0477064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0477129Z return func(*args, **kwargs) 2025-11-03T16:40:29.0477373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0477457Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0477460Z 2025-11-03T16:40:29.0477567Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0477775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0477842Z res = mod(**inputs) 2025-11-03T16:40:29.0478081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0478146Z outputs = self.model( 2025-11-03T16:40:29.0478389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0478458Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0478677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0478752Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0478998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0479075Z return func(*args, **kwargs) 2025-11-03T16:40:29.0479324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0479448Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0479451Z 2025-11-03T16:40:29.0479550Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0479744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0479805Z res = mod(**inputs) 2025-11-03T16:40:29.0480045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0480117Z outputs = self.model( 2025-11-03T16:40:29.0480372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0480449Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0480662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0480736Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0480986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0481068Z return func(*args, **kwargs) 2025-11-03T16:40:29.0481316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0481432Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0481640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0481713Z return self.act(input) 2025-11-03T16:40:29.0481717Z 2025-11-03T16:40:29.0481816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0482010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0482070Z res = mod(**inputs) 2025-11-03T16:40:29.0482316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0482380Z outputs = self.model( 2025-11-03T16:40:29.0482618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0482696Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0482912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0482996Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0483236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0483304Z return func(*args, **kwargs) 2025-11-03T16:40:29.0483583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0483665Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0483668Z 2025-11-03T16:40:29.0483773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0483977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0484040Z res = mod(**inputs) 2025-11-03T16:40:29.0484285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0484348Z outputs = self.model( 2025-11-03T16:40:29.0484595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0484680Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0484898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0484976Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0485212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0485289Z return func(*args, **kwargs) 2025-11-03T16:40:29.0485532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0485617Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0485620Z 2025-11-03T16:40:29.0485720Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0485912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0486352Z res = mod(**inputs) 2025-11-03T16:40:29.0486597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0486673Z outputs = self.model( 2025-11-03T16:40:29.0486912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0486982Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0487199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0487290Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0487531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0487598Z return func(*args, **kwargs) 2025-11-03T16:40:29.0487844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0487939Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0488170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0488242Z return func(*args, **kwargs) 2025-11-03T16:40:29.0488479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0488591Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0488596Z 2025-11-03T16:40:29.0488693Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0488882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0488960Z res = mod(**inputs) 2025-11-03T16:40:29.0489195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0489265Z outputs = self.model( 2025-11-03T16:40:29.0489497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0489587Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0489792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0489864Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0490093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0490158Z return func(*args, **kwargs) 2025-11-03T16:40:29.0490391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0490481Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0490705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0490793Z return func(*args, **kwargs) 2025-11-03T16:40:29.0491023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0491107Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0491110Z 2025-11-03T16:40:29.0491205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0491395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0491455Z res = mod(**inputs) 2025-11-03T16:40:29.0491683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0491753Z outputs = self.model( 2025-11-03T16:40:29.0491979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0492068Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0492277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0492352Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0492587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0492651Z return func(*args, **kwargs) 2025-11-03T16:40:29.0492890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0492996Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0493220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0493293Z return func(*args, **kwargs) 2025-11-03T16:40:29.0493523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0493634Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0493638Z 2025-11-03T16:40:29.0493732Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0493920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0493980Z res = mod(**inputs) 2025-11-03T16:40:29.0494207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0494279Z outputs = self.model( 2025-11-03T16:40:29.0494507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0494581Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0494786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0494859Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0495090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0495168Z return func(*args, **kwargs) 2025-11-03T16:40:29.0495401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0495491Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0495719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0495790Z return func(*args, **kwargs) 2025-11-03T16:40:29.0496018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0496152Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0496155Z 2025-11-03T16:40:29.0496267Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0496458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0496518Z res = mod(**inputs) 2025-11-03T16:40:29.0496748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0496817Z outputs = self.model( 2025-11-03T16:40:29.0497047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0497124Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0497330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0497403Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0497650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0497717Z return func(*args, **kwargs) 2025-11-03T16:40:29.0497954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0498047Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0498278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0498343Z return func(*args, **kwargs) 2025-11-03T16:40:29.0498586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0498673Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0498676Z 2025-11-03T16:40:29.0498771Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0498957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0499016Z res = mod(**inputs) 2025-11-03T16:40:29.0499247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0499318Z outputs = self.model( 2025-11-03T16:40:29.0499545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0499618Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0499824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0499898Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0500128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0500192Z return func(*args, **kwargs) 2025-11-03T16:40:29.0500432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0500523Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0500752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0500834Z return func(*args, **kwargs) 2025-11-03T16:40:29.0501065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0501163Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0501167Z 2025-11-03T16:40:29.0501262Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0501452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0501510Z res = mod(**inputs) 2025-11-03T16:40:29.0501740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0501811Z outputs = self.model( 2025-11-03T16:40:29.0502058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0502135Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0502341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0502413Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0502645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0502711Z return func(*args, **kwargs) 2025-11-03T16:40:29.0502946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0503036Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0503283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0503346Z return func(*args, **kwargs) 2025-11-03T16:40:29.0503574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0503699Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0503703Z 2025-11-03T16:40:29.0503796Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0503988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0504071Z res = mod(**inputs) 2025-11-03T16:40:29.0504302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0504374Z outputs = self.model( 2025-11-03T16:40:29.0504605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0504682Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0504890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0504971Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0505192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0505256Z return func(*args, **kwargs) 2025-11-03T16:40:29.0505496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0505587Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0505816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0505879Z return func(*args, **kwargs) 2025-11-03T16:40:29.0506109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0506192Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0506212Z 2025-11-03T16:40:29.0506306Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0506494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0506553Z res = mod(**inputs) 2025-11-03T16:40:29.0506784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0506857Z outputs = self.model( 2025-11-03T16:40:29.0507087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0507162Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0507368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0507467Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0507690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0507755Z return func(*args, **kwargs) 2025-11-03T16:40:29.0507989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0508103Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0508108Z 2025-11-03T16:40:29.0508207Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0508390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0508449Z res = mod(**inputs) 2025-11-03T16:40:29.0508683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0508761Z outputs = self.model( 2025-11-03T16:40:29.0508999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0509066Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0509278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0509349Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0509571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0509655Z return func(*args, **kwargs) 2025-11-03T16:40:29.0509890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0510008Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0510214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0510279Z return self.act(input) 2025-11-03T16:40:29.0510282Z 2025-11-03T16:40:29.0510388Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0510574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0510641Z res = mod(**inputs) 2025-11-03T16:40:29.0510878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0510941Z outputs = self.model( 2025-11-03T16:40:29.0511187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0511253Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0511471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0511546Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0511782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0511861Z return func(*args, **kwargs) 2025-11-03T16:40:29.0512091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0512174Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0512177Z 2025-11-03T16:40:29.0512271Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0512462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0512520Z res = mod(**inputs) 2025-11-03T16:40:29.0512746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0512815Z outputs = self.model( 2025-11-03T16:40:29.0513058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0513135Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0513464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0513540Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0513774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0513840Z return func(*args, **kwargs) 2025-11-03T16:40:29.0514081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0514212Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0514461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0514565Z return func(*args, **kwargs) 2025-11-03T16:40:29.0514813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0514933Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0514937Z 2025-11-03T16:40:29.0515037Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0515248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0515311Z res = mod(**inputs) 2025-11-03T16:40:29.0515572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0515645Z outputs = self.model( 2025-11-03T16:40:29.0515889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0515963Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0516173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0516253Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0516479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0516543Z return func(*args, **kwargs) 2025-11-03T16:40:29.0516783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0516877Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0517124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0517193Z return func(*args, **kwargs) 2025-11-03T16:40:29.0517439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0517528Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0517532Z 2025-11-03T16:40:29.0517631Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0517859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0517922Z res = mod(**inputs) 2025-11-03T16:40:29.0518169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0518253Z outputs = self.model( 2025-11-03T16:40:29.0518547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0518637Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0518903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0519002Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0519290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0519382Z return func(*args, **kwargs) 2025-11-03T16:40:29.0519670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0519770Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0520031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0520110Z return func(*args, **kwargs) 2025-11-03T16:40:29.0520376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0520493Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0520496Z 2025-11-03T16:40:29.0520599Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0520858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0520933Z res = mod(**inputs) 2025-11-03T16:40:29.0521233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0521311Z outputs = self.model( 2025-11-03T16:40:29.0521591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0521697Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0521949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0522045Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0522307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0522397Z return func(*args, **kwargs) 2025-11-03T16:40:29.0522689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0522788Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0523050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0523136Z return func(*args, **kwargs) 2025-11-03T16:40:29.0523409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0523564Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0523567Z 2025-11-03T16:40:29.0523670Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0523897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0523984Z res = mod(**inputs) 2025-11-03T16:40:29.0524277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0524353Z outputs = self.model( 2025-11-03T16:40:29.0524639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0524717Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0524966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0525049Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0525269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0525333Z return func(*args, **kwargs) 2025-11-03T16:40:29.0525566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0525657Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0525901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0525967Z return func(*args, **kwargs) 2025-11-03T16:40:29.0526203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0526283Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0526287Z 2025-11-03T16:40:29.0526383Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0526574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0526632Z res = mod(**inputs) 2025-11-03T16:40:29.0526867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0526928Z outputs = self.model( 2025-11-03T16:40:29.0527183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0527260Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0527469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0527550Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0527777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0527854Z return func(*args, **kwargs) 2025-11-03T16:40:29.0528089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0528179Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0528405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0528470Z return func(*args, **kwargs) 2025-11-03T16:40:29.0528706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0528796Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0528799Z 2025-11-03T16:40:29.0528892Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0529082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0529141Z res = mod(**inputs) 2025-11-03T16:40:29.0529377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0529438Z outputs = self.model( 2025-11-03T16:40:29.0529665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0529741Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0529948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0530028Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0530269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0530334Z return func(*args, **kwargs) 2025-11-03T16:40:29.0530572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0530667Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0530901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0530966Z return func(*args, **kwargs) 2025-11-03T16:40:29.0531204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0531339Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0531342Z 2025-11-03T16:40:29.0531437Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0531628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0531686Z res = mod(**inputs) 2025-11-03T16:40:29.0531921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0531986Z outputs = self.model( 2025-11-03T16:40:29.0532213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0532289Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0532492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0532572Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0532814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0532887Z return func(*args, **kwargs) 2025-11-03T16:40:29.0533118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0533210Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0533442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0533522Z return func(*args, **kwargs) 2025-11-03T16:40:29.0533760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0533838Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0533841Z 2025-11-03T16:40:29.0533938Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0534132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0534193Z res = mod(**inputs) 2025-11-03T16:40:29.0534433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0534497Z outputs = self.model( 2025-11-03T16:40:29.0534725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0534803Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0535006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0535086Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0535311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0535381Z return func(*args, **kwargs) 2025-11-03T16:40:29.0535615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0535740Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0535744Z 2025-11-03T16:40:29.0535845Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0536026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0536092Z res = mod(**inputs) 2025-11-03T16:40:29.0536320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0536381Z outputs = self.model( 2025-11-03T16:40:29.0536616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0536682Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0536909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0536983Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0537218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0537283Z return func(*args, **kwargs) 2025-11-03T16:40:29.0537512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0537631Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0537829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0537901Z return self.act(input) 2025-11-03T16:40:29.0537904Z 2025-11-03T16:40:29.0537999Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0538195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0538267Z res = mod(**inputs) 2025-11-03T16:40:29.0538499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0538571Z outputs = self.model( 2025-11-03T16:40:29.0538800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0538867Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0539097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0539172Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0539405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0539469Z return func(*args, **kwargs) 2025-11-03T16:40:29.0539706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0539785Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0539789Z 2025-11-03T16:40:29.0539884Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0540072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0540131Z res = mod(**inputs) 2025-11-03T16:40:29.0540364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0540428Z outputs = self.model( 2025-11-03T16:40:29.0540655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0540730Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0540932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0541014Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0541244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0541324Z return func(*args, **kwargs) 2025-11-03T16:40:29.0541562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0541638Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0541642Z 2025-11-03T16:40:29.0541744Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0541928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0541994Z res = mod(**inputs) 2025-11-03T16:40:29.0542228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0542291Z outputs = self.model( 2025-11-03T16:40:29.0542548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0542617Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0542828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0542901Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0543125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0543196Z return func(*args, **kwargs) 2025-11-03T16:40:29.0543431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0543530Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0543767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0543839Z return func(*args, **kwargs) 2025-11-03T16:40:29.0544070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0544175Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0544178Z 2025-11-03T16:40:29.0544278Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0544460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0544543Z res = mod(**inputs) 2025-11-03T16:40:29.0544772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0544834Z outputs = self.model( 2025-11-03T16:40:29.0545071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0545140Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0545353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0545427Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0545651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0545722Z return func(*args, **kwargs) 2025-11-03T16:40:29.0545952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0546052Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0546274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0546347Z return func(*args, **kwargs) 2025-11-03T16:40:29.0546578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0546653Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0546656Z 2025-11-03T16:40:29.0546773Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0546955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0547025Z res = mod(**inputs) 2025-11-03T16:40:29.0547252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0547317Z outputs = self.model( 2025-11-03T16:40:29.0547558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0547624Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0547833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0547906Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0548158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0548233Z return func(*args, **kwargs) 2025-11-03T16:40:29.0548460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0548558Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0548781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0548853Z return func(*args, **kwargs) 2025-11-03T16:40:29.0549079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0549180Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0549184Z 2025-11-03T16:40:29.0549301Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0549487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0549555Z res = mod(**inputs) 2025-11-03T16:40:29.0549785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0549847Z outputs = self.model( 2025-11-03T16:40:29.0550082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0550163Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0550380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0550454Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0550691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0550757Z return func(*args, **kwargs) 2025-11-03T16:40:29.0550990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0551090Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0551316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0551388Z return func(*args, **kwargs) 2025-11-03T16:40:29.0551622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0551750Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0551753Z 2025-11-03T16:40:29.0551856Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0552039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0552107Z res = mod(**inputs) 2025-11-03T16:40:29.0552340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0552428Z outputs = self.model( 2025-11-03T16:40:29.0552658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0552726Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0552943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0553017Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0553252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0553317Z return func(*args, **kwargs) 2025-11-03T16:40:29.0553552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0553669Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0553899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0553975Z return func(*args, **kwargs) 2025-11-03T16:40:29.0554277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0554372Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0554387Z 2025-11-03T16:40:29.0554488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0554675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0554745Z res = mod(**inputs) 2025-11-03T16:40:29.0554989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0555094Z outputs = self.model( 2025-11-03T16:40:29.0555343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0555414Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0555642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0555721Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0555976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0556055Z return func(*args, **kwargs) 2025-11-03T16:40:29.0556299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0556400Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0556642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0556716Z return func(*args, **kwargs) 2025-11-03T16:40:29.0556956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0557049Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0557060Z 2025-11-03T16:40:29.0557156Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0557347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0557416Z res = mod(**inputs) 2025-11-03T16:40:29.0557659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0557732Z outputs = self.model( 2025-11-03T16:40:29.0557977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0558046Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0558271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0558376Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0558617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0558682Z return func(*args, **kwargs) 2025-11-03T16:40:29.0558921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0559024Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0559256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0559329Z return func(*args, **kwargs) 2025-11-03T16:40:29.0559567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0559712Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0559719Z 2025-11-03T16:40:29.0559819Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0560006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0560076Z res = mod(**inputs) 2025-11-03T16:40:29.0560312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0560386Z outputs = self.model( 2025-11-03T16:40:29.0560626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0560694Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0560913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0561005Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0561246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0561314Z return func(*args, **kwargs) 2025-11-03T16:40:29.0561552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0561653Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0561902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0561975Z return func(*args, **kwargs) 2025-11-03T16:40:29.0562214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0562298Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0562302Z 2025-11-03T16:40:29.0562400Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0562589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0562668Z res = mod(**inputs) 2025-11-03T16:40:29.0562908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0562978Z outputs = self.model( 2025-11-03T16:40:29.0563218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0563288Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0563511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0563585Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0563825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0563897Z return func(*args, **kwargs) 2025-11-03T16:40:29.0564133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0564272Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0564276Z 2025-11-03T16:40:29.0564371Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0564564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0564626Z res = mod(**inputs) 2025-11-03T16:40:29.0564868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0564933Z outputs = self.model( 2025-11-03T16:40:29.0565171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0565247Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0565473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0565556Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0565787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0565851Z return func(*args, **kwargs) 2025-11-03T16:40:29.0566094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0566208Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0566422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0566489Z return self.act(input) 2025-11-03T16:40:29.0566492Z 2025-11-03T16:40:29.0566596Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0566799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0566862Z res = mod(**inputs) 2025-11-03T16:40:29.0567105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0567169Z outputs = self.model( 2025-11-03T16:40:29.0567412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0567498Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0567709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0567793Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0568025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0568097Z return func(*args, **kwargs) 2025-11-03T16:40:29.0568339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0568418Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0568429Z 2025-11-03T16:40:29.0568530Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0568717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0568785Z res = mod(**inputs) 2025-11-03T16:40:29.0569023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0569094Z outputs = self.model( 2025-11-03T16:40:29.0569334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0569401Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0583536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0583751Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0584170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0584258Z return func(*args, **kwargs) 2025-11-03T16:40:29.0584534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0584643Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0584907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0584980Z return func(*args, **kwargs) 2025-11-03T16:40:29.0585242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0585359Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0585365Z 2025-11-03T16:40:29.0585509Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0585721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0585790Z res = mod(**inputs) 2025-11-03T16:40:29.0586047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0586119Z outputs = self.model( 2025-11-03T16:40:29.0586369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0586448Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0586673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0586763Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0587028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0587106Z return func(*args, **kwargs) 2025-11-03T16:40:29.0587354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0587463Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0587702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0587794Z return func(*args, **kwargs) 2025-11-03T16:40:29.0588046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0588130Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0588134Z 2025-11-03T16:40:29.0588250Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0588451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0588518Z res = mod(**inputs) 2025-11-03T16:40:29.0588771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0588842Z outputs = self.model( 2025-11-03T16:40:29.0589093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0589167Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0589387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0589474Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0589711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0589788Z return func(*args, **kwargs) 2025-11-03T16:40:29.0590032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0590138Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0590394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0590462Z return func(*args, **kwargs) 2025-11-03T16:40:29.0590713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0590827Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0590831Z 2025-11-03T16:40:29.0590942Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0591136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0591199Z res = mod(**inputs) 2025-11-03T16:40:29.0591463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0591532Z outputs = self.model( 2025-11-03T16:40:29.0591777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0591852Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0592075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0592152Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0592388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0592465Z return func(*args, **kwargs) 2025-11-03T16:40:29.0592710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0592811Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0593057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0593124Z return func(*args, **kwargs) 2025-11-03T16:40:29.0593366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0593499Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0593502Z 2025-11-03T16:40:29.0593611Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0593811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0593875Z res = mod(**inputs) 2025-11-03T16:40:29.0594197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0594275Z outputs = self.model( 2025-11-03T16:40:29.0594526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0594595Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0594813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0594887Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0595130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0595200Z return func(*args, **kwargs) 2025-11-03T16:40:29.0595431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0595525Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0595751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0595815Z return func(*args, **kwargs) 2025-11-03T16:40:29.0596056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0596157Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0596161Z 2025-11-03T16:40:29.0596272Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0596453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0596517Z res = mod(**inputs) 2025-11-03T16:40:29.0596748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0596809Z outputs = self.model( 2025-11-03T16:40:29.0597046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0597111Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0597335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0597406Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0597635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0597705Z return func(*args, **kwargs) 2025-11-03T16:40:29.0597933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0598031Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0598254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0598317Z return func(*args, **kwargs) 2025-11-03T16:40:29.0598552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0598660Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0598666Z 2025-11-03T16:40:29.0598769Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0598955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0599022Z res = mod(**inputs) 2025-11-03T16:40:29.0599257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0599320Z outputs = self.model( 2025-11-03T16:40:29.0599575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0599643Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0599856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0599930Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0600155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0600228Z return func(*args, **kwargs) 2025-11-03T16:40:29.0600462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0600560Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0600784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0600849Z return func(*args, **kwargs) 2025-11-03T16:40:29.0601085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0601204Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0601208Z 2025-11-03T16:40:29.0601313Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0601502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0601571Z res = mod(**inputs) 2025-11-03T16:40:29.0601817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0601881Z outputs = self.model( 2025-11-03T16:40:29.0602118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0602190Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0602400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0602474Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0602697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0602769Z return func(*args, **kwargs) 2025-11-03T16:40:29.0603011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0603105Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0603333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0603401Z return func(*args, **kwargs) 2025-11-03T16:40:29.0603626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0603705Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0603708Z 2025-11-03T16:40:29.0603804Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0603986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0604047Z res = mod(**inputs) 2025-11-03T16:40:29.0604293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0604357Z outputs = self.model( 2025-11-03T16:40:29.0604597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0604664Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0604872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0604973Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0605195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0605265Z return func(*args, **kwargs) 2025-11-03T16:40:29.0605492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0605614Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0605617Z 2025-11-03T16:40:29.0605715Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0605902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0605967Z res = mod(**inputs) 2025-11-03T16:40:29.0606195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0606265Z outputs = self.model( 2025-11-03T16:40:29.0606507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0606579Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0606786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0606864Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0607102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0607171Z return func(*args, **kwargs) 2025-11-03T16:40:29.0607422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0607537Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0607750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0607819Z return self.act(input) 2025-11-03T16:40:29.0607823Z 2025-11-03T16:40:29.0607918Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0608106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0608166Z res = mod(**inputs) 2025-11-03T16:40:29.0608404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0608483Z outputs = self.model( 2025-11-03T16:40:29.0608717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0608796Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0609004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0609087Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0609319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0609383Z return func(*args, **kwargs) 2025-11-03T16:40:29.0609624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0609700Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0609704Z 2025-11-03T16:40:29.0609821Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0610008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0610078Z res = mod(**inputs) 2025-11-03T16:40:29.0610311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0610374Z outputs = self.model( 2025-11-03T16:40:29.0610612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0610692Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0610905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0610977Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0611204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0611275Z return func(*args, **kwargs) 2025-11-03T16:40:29.0611508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0611585Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0611588Z 2025-11-03T16:40:29.0611681Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0611867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0611928Z res = mod(**inputs) 2025-11-03T16:40:29.0612160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0612228Z outputs = self.model( 2025-11-03T16:40:29.0612459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0612530Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0612740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0612809Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0613053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0613114Z return func(*args, **kwargs) 2025-11-03T16:40:29.0613490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0613590Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0613809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0613877Z return func(*args, **kwargs) 2025-11-03T16:40:29.0614105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0614268Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0614272Z 2025-11-03T16:40:29.0614369Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0614560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0614621Z res = mod(**inputs) 2025-11-03T16:40:29.0614851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0614924Z outputs = self.model( 2025-11-03T16:40:29.0615154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0615230Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0615437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0615539Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0615774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0615843Z return func(*args, **kwargs) 2025-11-03T16:40:29.0616080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0616174Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0616398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0616495Z return func(*args, **kwargs) 2025-11-03T16:40:29.0616728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0616810Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0616813Z 2025-11-03T16:40:29.0616905Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0617096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0617153Z res = mod(**inputs) 2025-11-03T16:40:29.0617391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0617459Z outputs = self.model( 2025-11-03T16:40:29.0617690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0617762Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0617967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0618037Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0618266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0618329Z return func(*args, **kwargs) 2025-11-03T16:40:29.0618570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0618682Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0618907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0618972Z return func(*args, **kwargs) 2025-11-03T16:40:29.0619199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0619306Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0619309Z 2025-11-03T16:40:29.0619403Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0619590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0619646Z res = mod(**inputs) 2025-11-03T16:40:29.0619888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0619956Z outputs = self.model( 2025-11-03T16:40:29.0620186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0620254Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0620457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0620531Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0620756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0620817Z return func(*args, **kwargs) 2025-11-03T16:40:29.0621046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0621150Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0621384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0621446Z return func(*args, **kwargs) 2025-11-03T16:40:29.0621676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0621809Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0621827Z 2025-11-03T16:40:29.0621922Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0622104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0622161Z res = mod(**inputs) 2025-11-03T16:40:29.0622391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0622456Z outputs = self.model( 2025-11-03T16:40:29.0622683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0622757Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0622959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0623030Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0623260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0623325Z return func(*args, **kwargs) 2025-11-03T16:40:29.0623564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0623653Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0623891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0623958Z return func(*args, **kwargs) 2025-11-03T16:40:29.0624187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0624293Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0624297Z 2025-11-03T16:40:29.0624392Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0624582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0624644Z res = mod(**inputs) 2025-11-03T16:40:29.0624872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0624941Z outputs = self.model( 2025-11-03T16:40:29.0625168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0625243Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0625465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0625547Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0625773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0625837Z return func(*args, **kwargs) 2025-11-03T16:40:29.0626078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0626169Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0626399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0626462Z return func(*args, **kwargs) 2025-11-03T16:40:29.0626715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0626814Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0626817Z 2025-11-03T16:40:29.0626911Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0627095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0627152Z res = mod(**inputs) 2025-11-03T16:40:29.0627383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0627471Z outputs = self.model( 2025-11-03T16:40:29.0627699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0627775Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0627980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0628067Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0628295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0628362Z return func(*args, **kwargs) 2025-11-03T16:40:29.0628601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0628692Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0628923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0628989Z return func(*args, **kwargs) 2025-11-03T16:40:29.0629219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0629347Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0629351Z 2025-11-03T16:40:29.0629448Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0629640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0629718Z res = mod(**inputs) 2025-11-03T16:40:29.0629962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0630027Z outputs = self.model( 2025-11-03T16:40:29.0630262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0630341Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0630551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0630632Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0630859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0630927Z return func(*args, **kwargs) 2025-11-03T16:40:29.0631180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0631290Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0631522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0631588Z return func(*args, **kwargs) 2025-11-03T16:40:29.0631829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0631908Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0631911Z 2025-11-03T16:40:29.0632015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0632203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0632264Z res = mod(**inputs) 2025-11-03T16:40:29.0632522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0632589Z outputs = self.model( 2025-11-03T16:40:29.0632833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0632901Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0633113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0633210Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0633438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0633511Z return func(*args, **kwargs) 2025-11-03T16:40:29.0633746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0633869Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0633872Z 2025-11-03T16:40:29.0633970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0634233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0634313Z res = mod(**inputs) 2025-11-03T16:40:29.0634555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0634631Z outputs = self.model( 2025-11-03T16:40:29.0634874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0634945Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0635210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0635298Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0635542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0635629Z return func(*args, **kwargs) 2025-11-03T16:40:29.0635865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0635987Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0636191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0636269Z return self.act(input) 2025-11-03T16:40:29.0636272Z 2025-11-03T16:40:29.0636369Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0636566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0636627Z res = mod(**inputs) 2025-11-03T16:40:29.0636874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0636949Z outputs = self.model( 2025-11-03T16:40:29.0637191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0637267Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0637480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0637554Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0637809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0637873Z return func(*args, **kwargs) 2025-11-03T16:40:29.0638121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0638200Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0638218Z 2025-11-03T16:40:29.0638330Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0638518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0638585Z res = mod(**inputs) 2025-11-03T16:40:29.0638820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0638883Z outputs = self.model( 2025-11-03T16:40:29.0639128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0639214Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0639434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0639508Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0639745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0639816Z return func(*args, **kwargs) 2025-11-03T16:40:29.0640053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0640155Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0640388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0640462Z return func(*args, **kwargs) 2025-11-03T16:40:29.0640697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0640803Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0640806Z 2025-11-03T16:40:29.0640912Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0641099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0641168Z res = mod(**inputs) 2025-11-03T16:40:29.0641405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0641486Z outputs = self.model( 2025-11-03T16:40:29.0641730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0641799Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0642020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0642093Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0642321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0642392Z return func(*args, **kwargs) 2025-11-03T16:40:29.0642642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0642744Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0642975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0643045Z return func(*args, **kwargs) 2025-11-03T16:40:29.0643284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0643364Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0643367Z 2025-11-03T16:40:29.0643472Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0643657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0643723Z res = mod(**inputs) 2025-11-03T16:40:29.0643978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0644044Z outputs = self.model( 2025-11-03T16:40:29.0644285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0644353Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0644571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0644645Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0644899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0644964Z return func(*args, **kwargs) 2025-11-03T16:40:29.0645201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0645299Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0645534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0645614Z return func(*args, **kwargs) 2025-11-03T16:40:29.0645844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0645945Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0645949Z 2025-11-03T16:40:29.0646049Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0646232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0646297Z res = mod(**inputs) 2025-11-03T16:40:29.0646526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0646588Z outputs = self.model( 2025-11-03T16:40:29.0646826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0646894Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0647105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0647195Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0647431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0647495Z return func(*args, **kwargs) 2025-11-03T16:40:29.0647723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0647819Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0648040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0648110Z return func(*args, **kwargs) 2025-11-03T16:40:29.0648367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0648493Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0648504Z 2025-11-03T16:40:29.0648599Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0648781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0648847Z res = mod(**inputs) 2025-11-03T16:40:29.0649080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0649149Z outputs = self.model( 2025-11-03T16:40:29.0649380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0649447Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0649678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0649752Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0649986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0650051Z return func(*args, **kwargs) 2025-11-03T16:40:29.0650280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0650394Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0650620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0650690Z return func(*args, **kwargs) 2025-11-03T16:40:29.0650920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0651001Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0651011Z 2025-11-03T16:40:29.0651106Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0651287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0651354Z res = mod(**inputs) 2025-11-03T16:40:29.0651586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0651655Z outputs = self.model( 2025-11-03T16:40:29.0651887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0651955Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0652175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0652247Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0652479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0652543Z return func(*args, **kwargs) 2025-11-03T16:40:29.0652787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0652885Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0653110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0653183Z return func(*args, **kwargs) 2025-11-03T16:40:29.0653412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0653508Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0653512Z 2025-11-03T16:40:29.0653605Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0653790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0653871Z res = mod(**inputs) 2025-11-03T16:40:29.0654101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0654172Z outputs = self.model( 2025-11-03T16:40:29.0654406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0654473Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0654686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0654758Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0654987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0655051Z return func(*args, **kwargs) 2025-11-03T16:40:29.0655293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0655391Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0655614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0655684Z return func(*args, **kwargs) 2025-11-03T16:40:29.0655912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0656051Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0656054Z 2025-11-03T16:40:29.0656146Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0656329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0656395Z res = mod(**inputs) 2025-11-03T16:40:29.0656629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0656697Z outputs = self.model( 2025-11-03T16:40:29.0656925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0656992Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0657203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0657275Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0657506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0657568Z return func(*args, **kwargs) 2025-11-03T16:40:29.0657804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0657894Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0658120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0658205Z return func(*args, **kwargs) 2025-11-03T16:40:29.0658438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0658523Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0658526Z 2025-11-03T16:40:29.0658620Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0658808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0658875Z res = mod(**inputs) 2025-11-03T16:40:29.0659110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0659180Z outputs = self.model( 2025-11-03T16:40:29.0659429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0659496Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0659707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0659780Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0660014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0660079Z return func(*args, **kwargs) 2025-11-03T16:40:29.0660314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0660425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0660428Z 2025-11-03T16:40:29.0660521Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0660732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0660795Z res = mod(**inputs) 2025-11-03T16:40:29.0661042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0661107Z outputs = self.model( 2025-11-03T16:40:29.0661341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0661414Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0661658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0661737Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0661959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0662023Z return func(*args, **kwargs) 2025-11-03T16:40:29.0662303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0662416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0662630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0662697Z return self.act(input) 2025-11-03T16:40:29.0662700Z 2025-11-03T16:40:29.0662803Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0662988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0663050Z res = mod(**inputs) 2025-11-03T16:40:29.0663294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0663358Z outputs = self.model( 2025-11-03T16:40:29.0663599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0663669Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0663878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0663976Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0664205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0664276Z return func(*args, **kwargs) 2025-11-03T16:40:29.0664510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0664594Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0664598Z 2025-11-03T16:40:29.0664694Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0664879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0664945Z res = mod(**inputs) 2025-11-03T16:40:29.0665197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0665271Z outputs = self.model( 2025-11-03T16:40:29.0665507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0665575Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0665791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0665867Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0666107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0666172Z return func(*args, **kwargs) 2025-11-03T16:40:29.0666406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-11-03T16:40:29.0666507Z hidden_states = residual + hidden_states 2025-11-03T16:40:29.0666513Z 2025-11-03T16:40:29.0666611Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0666803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0666863Z res = mod(**inputs) 2025-11-03T16:40:29.0667108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0667189Z outputs = self.model( 2025-11-03T16:40:29.0667431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0667508Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0667722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0667803Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0668042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0668107Z return func(*args, **kwargs) 2025-11-03T16:40:29.0668355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0668449Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0668693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0668758Z return func(*args, **kwargs) 2025-11-03T16:40:29.0668998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0669110Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0669114Z 2025-11-03T16:40:29.0669212Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0669410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0669471Z res = mod(**inputs) 2025-11-03T16:40:29.0669739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0669802Z outputs = self.model( 2025-11-03T16:40:29.0670034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0670110Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0670322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0670402Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0670632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0670695Z return func(*args, **kwargs) 2025-11-03T16:40:29.0670952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0671046Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0671280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0671343Z return func(*args, **kwargs) 2025-11-03T16:40:29.0671584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-11-03T16:40:29.0671660Z key_states = self.k_proj(current_states) 2025-11-03T16:40:29.0671663Z 2025-11-03T16:40:29.0671760Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0671955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0672016Z res = mod(**inputs) 2025-11-03T16:40:29.0672272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0672339Z outputs = self.model( 2025-11-03T16:40:29.0672574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0672647Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0672857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0672952Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0673181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0673244Z return func(*args, **kwargs) 2025-11-03T16:40:29.0673483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0673576Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0673813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0673880Z return func(*args, **kwargs) 2025-11-03T16:40:29.0674213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 158, in forward 2025-11-03T16:40:29.0674326Z query_states = self.q_proj(hidden_states) * self.scaling 2025-11-03T16:40:29.0674330Z 2025-11-03T16:40:29.0674432Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0674629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0674692Z res = mod(**inputs) 2025-11-03T16:40:29.0674941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0675006Z outputs = self.model( 2025-11-03T16:40:29.0675251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0675330Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0675569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0675654Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0675905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0675972Z return func(*args, **kwargs) 2025-11-03T16:40:29.0676225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0676319Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0676568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0676637Z return func(*args, **kwargs) 2025-11-03T16:40:29.0676915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-11-03T16:40:29.0677046Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-11-03T16:40:29.0677050Z 2025-11-03T16:40:29.0677148Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0677344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0677407Z res = mod(**inputs) 2025-11-03T16:40:29.0677650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0677713Z outputs = self.model( 2025-11-03T16:40:29.0677948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0678023Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0678250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0678335Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0678564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0678640Z return func(*args, **kwargs) 2025-11-03T16:40:29.0678876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0678984Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0679223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0679286Z return func(*args, **kwargs) 2025-11-03T16:40:29.0679531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-11-03T16:40:29.0679614Z value_states = self.v_proj(current_states) 2025-11-03T16:40:29.0679617Z 2025-11-03T16:40:29.0679715Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0679909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0679970Z res = mod(**inputs) 2025-11-03T16:40:29.0680215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0680281Z outputs = self.model( 2025-11-03T16:40:29.0680520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0680589Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0680799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0680881Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0681112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0681200Z return func(*args, **kwargs) 2025-11-03T16:40:29.0681435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0681527Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0681764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0681829Z return func(*args, **kwargs) 2025-11-03T16:40:29.0682069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-11-03T16:40:29.0682157Z attn_output = torch.bmm(attn_probs, value_states) 2025-11-03T16:40:29.0682160Z 2025-11-03T16:40:29.0682254Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0682462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0682523Z res = mod(**inputs) 2025-11-03T16:40:29.0682771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0682836Z outputs = self.model( 2025-11-03T16:40:29.0683080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0683149Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0683360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0683440Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0683670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0683755Z return func(*args, **kwargs) 2025-11-03T16:40:29.0683991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0684083Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0684317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0684382Z return func(*args, **kwargs) 2025-11-03T16:40:29.0684621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-11-03T16:40:29.0684759Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-11-03T16:40:29.0684763Z 2025-11-03T16:40:29.0684866Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0685049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0685108Z res = mod(**inputs) 2025-11-03T16:40:29.0685349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0685414Z outputs = self.model( 2025-11-03T16:40:29.0685652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0685719Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0685928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0686010Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0686238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0686306Z return func(*args, **kwargs) 2025-11-03T16:40:29.0686537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-11-03T16:40:29.0686631Z hidden_states, self_attn_weights = self.self_attn( 2025-11-03T16:40:29.0686866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0686948Z return func(*args, **kwargs) 2025-11-03T16:40:29.0687183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-11-03T16:40:29.0687259Z attn_output = self.out_proj(attn_output) 2025-11-03T16:40:29.0687263Z 2025-11-03T16:40:29.0687366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0687550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0687609Z res = mod(**inputs) 2025-11-03T16:40:29.0687849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0687913Z outputs = self.model( 2025-11-03T16:40:29.0688169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0688250Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0688455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0688534Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0688760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0688833Z return func(*args, **kwargs) 2025-11-03T16:40:29.0689064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0689177Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0689188Z 2025-11-03T16:40:29.0689284Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0689488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0689555Z res = mod(**inputs) 2025-11-03T16:40:29.0689786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0689857Z outputs = self.model( 2025-11-03T16:40:29.0690085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0690176Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0690388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0690461Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0690687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0690750Z return func(*args, **kwargs) 2025-11-03T16:40:29.0690980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-11-03T16:40:29.0691099Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-11-03T16:40:29.0691297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:40:29.0691368Z return self.act(input) 2025-11-03T16:40:29.0691371Z 2025-11-03T16:40:29.0691465Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0691654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0691711Z res = mod(**inputs) 2025-11-03T16:40:29.0691938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-11-03T16:40:29.0692008Z outputs = self.model( 2025-11-03T16:40:29.0692238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-11-03T16:40:29.0692312Z layer_outputs = decoder_layer( 2025-11-03T16:40:29.0692520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:40:29.0692609Z return super().__call__(*args, **kwargs) 2025-11-03T16:40:29.0692840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-11-03T16:40:29.0692904Z return func(*args, **kwargs) 2025-11-03T16:40:29.0693141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-11-03T16:40:29.0693217Z hidden_states = self.fc2(hidden_states) 2025-11-03T16:40:29.0693220Z 2025-11-03T16:40:29.0693315Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0693504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0693564Z res = mod(**inputs) 2025-11-03T16:40:29.0693814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-11-03T16:40:29.0693891Z logits = self.lm_head(outputs[0]) 2025-11-03T16:40:29.0693894Z 2025-11-03T16:40:29.0693994Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:40:29.0694172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:40:29.0694233Z res = mod(**inputs) 2025-11-03T16:40:29.0694468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 689, in forward 2025-11-03T16:40:29.0694536Z loss = self.loss_function( 2025-11-03T16:40:29.0694765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-11-03T16:40:29.0694939Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-11-03T16:40:29.0695175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-11-03T16:40:29.0695365Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-11-03T16:40:29.0695369Z 2025-11-03T16:40:40.9153411Z Compilation time (from dynamo_timed): 24.599694396 2025-11-03T16:40:40.9247043Z pass 2025-11-03T16:40:40.9247538Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:40.9249484Z TIMING: _recursive_pre_grad_passes:0.01176 _recursive_joint_graph_passes:0.76318 _recursive_post_grad_passes:0.2324 async_compile.wait:0.68762 code_gen:11.23892 inductor_compile:14.43785 backend_compile:20.07486 gc:0.00234 entire_frame_compile:24.59969 total_wall_time:24.59969 2025-11-03T16:40:40.9250443Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:16773 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:4833 2025-11-03T16:40:40.9250912Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-11-03T16:40:43.4867814Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:40:43.4869116Z import pynvml # type: ignore[import] 2025-11-03T16:40:46.6477986Z 2025-11-03T16:40:49.2996195Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:40:49.2998613Z loading model: 0it [00:02, ?it/s] 2025-11-03T16:40:49.3016181Z cpu eval XLNetLMHeadModel 2025-11-03T16:40:51.8832855Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:52.8019928Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:40:53.7317124Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:41:13.9703129Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9705041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9705506Z res = mod(**inputs) 2025-11-03T16:41:13.9711385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9713673Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9714421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-11-03T16:41:13.9719231Z word_emb_k = self.word_embedding(input_ids) 2025-11-03T16:41:13.9721244Z 2025-11-03T16:41:13.9728949Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9730108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9730632Z res = mod(**inputs) 2025-11-03T16:41:13.9731057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9731470Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9731990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-11-03T16:41:13.9732436Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-11-03T16:41:13.9732925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-11-03T16:41:13.9733410Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-11-03T16:41:13.9733936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-11-03T16:41:13.9734425Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-11-03T16:41:13.9734639Z 2025-11-03T16:41:13.9734746Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9735105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9735427Z res = mod(**inputs) 2025-11-03T16:41:13.9735788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9736220Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9736593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-11-03T16:41:13.9737002Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-11-03T16:41:13.9737463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-11-03T16:41:13.9737970Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-11-03T16:41:13.9738424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-11-03T16:41:13.9738903Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-11-03T16:41:13.9739113Z 2025-11-03T16:41:13.9739219Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9739578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9739895Z res = mod(**inputs) 2025-11-03T16:41:13.9740299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9740683Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9741069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9741493Z outputs = layer_module( 2025-11-03T16:41:13.9741844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9742216Z outputs = self.rel_attn( 2025-11-03T16:41:13.9742583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9742978Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9743387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9743816Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9743992Z 2025-11-03T16:41:13.9744095Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9744498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9744829Z res = mod(**inputs) 2025-11-03T16:41:13.9745205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9745592Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9745977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9746355Z outputs = layer_module( 2025-11-03T16:41:13.9746716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9747081Z outputs = self.rel_attn( 2025-11-03T16:41:13.9747450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9747900Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9748308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9748734Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9748893Z 2025-11-03T16:41:13.9748998Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9749352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9749684Z res = mod(**inputs) 2025-11-03T16:41:13.9750039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9750441Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9750824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9751201Z outputs = layer_module( 2025-11-03T16:41:13.9751564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9751937Z outputs = self.rel_attn( 2025-11-03T16:41:13.9752302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9752686Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9753097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9753533Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9753700Z 2025-11-03T16:41:13.9753815Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9754325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9754679Z res = mod(**inputs) 2025-11-03T16:41:13.9755076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9755562Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9755955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9756324Z outputs = layer_module( 2025-11-03T16:41:13.9756704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9757077Z outputs = self.rel_attn( 2025-11-03T16:41:13.9757436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9757817Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9758227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9758679Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9758850Z 2025-11-03T16:41:13.9758955Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9759305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9759609Z res = mod(**inputs) 2025-11-03T16:41:13.9759962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9760355Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9760738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9761112Z outputs = layer_module( 2025-11-03T16:41:13.9761485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9761878Z outputs = self.rel_attn( 2025-11-03T16:41:13.9762244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9762635Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9763049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9763478Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9763661Z 2025-11-03T16:41:13.9763771Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9764121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9764426Z res = mod(**inputs) 2025-11-03T16:41:13.9764781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9765175Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9765567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9765936Z outputs = layer_module( 2025-11-03T16:41:13.9766308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9766684Z outputs = self.rel_attn( 2025-11-03T16:41:13.9767061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9767460Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9767867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9768314Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9768496Z 2025-11-03T16:41:13.9768601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9768955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9769321Z res = mod(**inputs) 2025-11-03T16:41:13.9769669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9770061Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9770447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9770826Z outputs = layer_module( 2025-11-03T16:41:13.9771190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9771582Z outputs = self.rel_attn( 2025-11-03T16:41:13.9771945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9772350Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9772771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9773213Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9773390Z 2025-11-03T16:41:13.9773496Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9773881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9774213Z res = mod(**inputs) 2025-11-03T16:41:13.9774583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9774980Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9775400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9775897Z outputs = layer_module( 2025-11-03T16:41:13.9776272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9776664Z outputs = self.rel_attn( 2025-11-03T16:41:13.9777041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9777468Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9777934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9778415Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9778594Z 2025-11-03T16:41:13.9778711Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9779089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9779419Z res = mod(**inputs) 2025-11-03T16:41:13.9779789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9780200Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9780609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9781008Z outputs = layer_module( 2025-11-03T16:41:13.9781389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9781781Z outputs = self.rel_attn( 2025-11-03T16:41:13.9782162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9782567Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9783003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9783451Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9783634Z 2025-11-03T16:41:13.9783748Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9784110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9784441Z res = mod(**inputs) 2025-11-03T16:41:13.9784839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9785249Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9785652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9786032Z outputs = layer_module( 2025-11-03T16:41:13.9786408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9786818Z outputs = self.rel_attn( 2025-11-03T16:41:13.9787193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9787593Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9788009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9788457Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9788635Z 2025-11-03T16:41:13.9788744Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9789113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9789437Z res = mod(**inputs) 2025-11-03T16:41:13.9789816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9790228Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9790652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9791064Z outputs = layer_module( 2025-11-03T16:41:13.9791452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9791859Z outputs = self.rel_attn( 2025-11-03T16:41:13.9792274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9792684Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9793110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9793564Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9793752Z 2025-11-03T16:41:13.9793865Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9794339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9794694Z res = mod(**inputs) 2025-11-03T16:41:13.9795080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9795506Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9795937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9796348Z outputs = layer_module( 2025-11-03T16:41:13.9796742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9797144Z outputs = self.rel_attn( 2025-11-03T16:41:13.9797543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9797974Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9798454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9798941Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9799120Z 2025-11-03T16:41:13.9799233Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9799620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9799964Z res = mod(**inputs) 2025-11-03T16:41:13.9800349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9800779Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9801203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9801642Z outputs = layer_module( 2025-11-03T16:41:13.9802019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9802427Z outputs = self.rel_attn( 2025-11-03T16:41:13.9802818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9803235Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9803658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9804108Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9804274Z 2025-11-03T16:41:13.9804387Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9804755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9805081Z res = mod(**inputs) 2025-11-03T16:41:13.9805439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9805847Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9806248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9806626Z outputs = layer_module( 2025-11-03T16:41:13.9807023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9807409Z outputs = self.rel_attn( 2025-11-03T16:41:13.9807791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9808198Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9808625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9809073Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9809248Z 2025-11-03T16:41:13.9809352Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9809717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9810032Z res = mod(**inputs) 2025-11-03T16:41:13.9810396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9810802Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9811200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9811595Z outputs = layer_module( 2025-11-03T16:41:13.9811963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9812356Z outputs = self.rel_attn( 2025-11-03T16:41:13.9812747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9813156Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9813955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9814407Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9814583Z 2025-11-03T16:41:13.9814691Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9815059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9815385Z res = mod(**inputs) 2025-11-03T16:41:13.9815746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9816205Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9816612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9817001Z outputs = layer_module( 2025-11-03T16:41:13.9817376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9817757Z outputs = self.rel_attn( 2025-11-03T16:41:13.9818137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9818539Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9818972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9819449Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9819618Z 2025-11-03T16:41:13.9819731Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9820122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9820481Z res = mod(**inputs) 2025-11-03T16:41:13.9820882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9821355Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9821793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9822174Z outputs = layer_module( 2025-11-03T16:41:13.9822548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9822934Z outputs = self.rel_attn( 2025-11-03T16:41:13.9823303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9823707Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9824143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9824590Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9824753Z 2025-11-03T16:41:13.9824864Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9825220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9825554Z res = mod(**inputs) 2025-11-03T16:41:13.9825906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9826304Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9826687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9827060Z outputs = layer_module( 2025-11-03T16:41:13.9827454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9827830Z outputs = self.rel_attn( 2025-11-03T16:41:13.9828193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9828582Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9829000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9829444Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9829610Z 2025-11-03T16:41:13.9829722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9830107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9830425Z res = mod(**inputs) 2025-11-03T16:41:13.9830788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9831215Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9831643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9832047Z outputs = layer_module( 2025-11-03T16:41:13.9832450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9832862Z outputs = self.rel_attn( 2025-11-03T16:41:13.9833255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9833682Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9834217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9834723Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9834905Z 2025-11-03T16:41:13.9835016Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9835398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9835742Z res = mod(**inputs) 2025-11-03T16:41:13.9836127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9836526Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9836917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9837298Z outputs = layer_module( 2025-11-03T16:41:13.9838262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9838638Z outputs = self.rel_attn( 2025-11-03T16:41:13.9839013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9839411Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9839828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9840258Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9840429Z 2025-11-03T16:41:13.9840534Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9840886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9841203Z res = mod(**inputs) 2025-11-03T16:41:13.9841568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9841957Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9842372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9842749Z outputs = layer_module( 2025-11-03T16:41:13.9843120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9843502Z outputs = self.rel_attn( 2025-11-03T16:41:13.9843862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9844258Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9844676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9846016Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9846185Z 2025-11-03T16:41:13.9846296Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9846649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9846970Z res = mod(**inputs) 2025-11-03T16:41:13.9847331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9847739Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9848118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9848491Z outputs = layer_module( 2025-11-03T16:41:13.9848849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9849226Z outputs = self.rel_attn( 2025-11-03T16:41:13.9849605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9849989Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9850401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9850837Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9850999Z 2025-11-03T16:41:13.9851127Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9851477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9851780Z res = mod(**inputs) 2025-11-03T16:41:13.9852133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9852525Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9852924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9853295Z outputs = layer_module( 2025-11-03T16:41:13.9853654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9854032Z outputs = self.rel_attn( 2025-11-03T16:41:13.9854397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9854790Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9855191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9855623Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9855792Z 2025-11-03T16:41:13.9855893Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9856247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9856568Z res = mod(**inputs) 2025-11-03T16:41:13.9856946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9857348Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9857748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9858135Z outputs = layer_module( 2025-11-03T16:41:13.9858497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9858883Z outputs = self.rel_attn( 2025-11-03T16:41:13.9859255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9859663Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9860115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9860554Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9860730Z 2025-11-03T16:41:13.9860835Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9861200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9861524Z res = mod(**inputs) 2025-11-03T16:41:13.9861889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9862284Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9862699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9863111Z outputs = layer_module( 2025-11-03T16:41:13.9863487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9863871Z outputs = self.rel_attn( 2025-11-03T16:41:13.9864243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:13.9864662Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:13.9864817Z 2025-11-03T16:41:13.9864944Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9865303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9865618Z res = mod(**inputs) 2025-11-03T16:41:13.9865976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9866377Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9866779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9867163Z outputs = layer_module( 2025-11-03T16:41:13.9867529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9867918Z outputs = self.rel_attn( 2025-11-03T16:41:13.9868295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:13.9868711Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:13.9868867Z 2025-11-03T16:41:13.9868972Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9869328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9869650Z res = mod(**inputs) 2025-11-03T16:41:13.9870014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9870418Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9870834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9871224Z outputs = layer_module( 2025-11-03T16:41:13.9871599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9871992Z outputs = self.rel_attn( 2025-11-03T16:41:13.9872362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9872761Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9873174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:13.9873647Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:13.9873853Z 2025-11-03T16:41:13.9873969Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9874420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9874765Z res = mod(**inputs) 2025-11-03T16:41:13.9875161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9875597Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9876029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-11-03T16:41:13.9876501Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-11-03T16:41:13.9877011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-11-03T16:41:13.9877540Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-11-03T16:41:13.9878023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-11-03T16:41:13.9878536Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-11-03T16:41:13.9878743Z 2025-11-03T16:41:13.9878850Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9879213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9879555Z res = mod(**inputs) 2025-11-03T16:41:13.9879921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9880333Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9880729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9881122Z outputs = layer_module( 2025-11-03T16:41:13.9881484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9881860Z outputs = self.rel_attn( 2025-11-03T16:41:13.9882218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:13.9882657Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:13.9882870Z 2025-11-03T16:41:13.9882981Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9883341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9883656Z res = mod(**inputs) 2025-11-03T16:41:13.9884001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9884391Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9884785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9885184Z outputs = layer_module( 2025-11-03T16:41:13.9885549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9885915Z outputs = self.rel_attn( 2025-11-03T16:41:13.9886284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9886673Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9887068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:13.9887511Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:13.9887700Z 2025-11-03T16:41:13.9887802Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9888173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9888491Z res = mod(**inputs) 2025-11-03T16:41:13.9888845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9889227Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9889619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9889994Z outputs = layer_module( 2025-11-03T16:41:13.9890356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9890732Z outputs = self.rel_attn( 2025-11-03T16:41:13.9891101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:13.9891510Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:13.9891664Z 2025-11-03T16:41:13.9891764Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9892115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9892413Z res = mod(**inputs) 2025-11-03T16:41:13.9892750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9893135Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9893510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9893878Z outputs = layer_module( 2025-11-03T16:41:13.9894221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9894590Z outputs = self.rel_attn( 2025-11-03T16:41:13.9894943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9895315Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9895701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:13.9896128Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:13.9896303Z 2025-11-03T16:41:13.9896415Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9896748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9897045Z res = mod(**inputs) 2025-11-03T16:41:13.9897377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9897751Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9898123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9898499Z outputs = layer_module( 2025-11-03T16:41:13.9898849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9899201Z outputs = self.rel_attn( 2025-11-03T16:41:13.9899554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9899947Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9900355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9900781Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9900942Z 2025-11-03T16:41:13.9901044Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9901409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9901716Z res = mod(**inputs) 2025-11-03T16:41:13.9902056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9902434Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9902822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9903191Z outputs = layer_module( 2025-11-03T16:41:13.9903541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9903909Z outputs = self.rel_attn( 2025-11-03T16:41:13.9904268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9904673Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9905085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9905517Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9905681Z 2025-11-03T16:41:13.9905790Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9906129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9906463Z res = mod(**inputs) 2025-11-03T16:41:13.9906804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9907190Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9907560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9907929Z outputs = layer_module( 2025-11-03T16:41:13.9908283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:13.9908797Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:13.9909305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:13.9909683Z return forward_fn(*input_tensors) 2025-11-03T16:41:13.9910059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:13.9910432Z output_x = self.ff(output_x) 2025-11-03T16:41:13.9910793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:13.9911165Z output = self.layer_1(output) 2025-11-03T16:41:13.9911286Z 2025-11-03T16:41:13.9911388Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9911734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9912070Z res = mod(**inputs) 2025-11-03T16:41:13.9912420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9912804Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9913178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9913697Z outputs = layer_module( 2025-11-03T16:41:13.9914094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:13.9914684Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:13.9915284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:13.9915669Z return forward_fn(*input_tensors) 2025-11-03T16:41:13.9916051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:13.9916424Z output_x = self.ff(output_x) 2025-11-03T16:41:13.9916784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:13.9917166Z output = self.activation_function(output) 2025-11-03T16:41:13.9917514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:13.9917849Z return self.act(input) 2025-11-03T16:41:13.9917957Z 2025-11-03T16:41:13.9918065Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9918442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9918747Z res = mod(**inputs) 2025-11-03T16:41:13.9919095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9919481Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9919860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9920248Z outputs = layer_module( 2025-11-03T16:41:13.9920608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:13.9921109Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:13.9921615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:13.9922000Z return forward_fn(*input_tensors) 2025-11-03T16:41:13.9922364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:13.9922742Z output_x = self.ff(output_x) 2025-11-03T16:41:13.9923107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:13.9923477Z output = self.layer_2(output) 2025-11-03T16:41:13.9923601Z 2025-11-03T16:41:13.9923709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9924049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9924366Z res = mod(**inputs) 2025-11-03T16:41:13.9924708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9925094Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9925469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9925844Z outputs = layer_module( 2025-11-03T16:41:13.9926194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9926555Z outputs = self.rel_attn( 2025-11-03T16:41:13.9926903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:13.9927281Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:13.9927432Z 2025-11-03T16:41:13.9927528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9927860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9928158Z res = mod(**inputs) 2025-11-03T16:41:13.9928510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9928884Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9929256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9929611Z outputs = layer_module( 2025-11-03T16:41:13.9929955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9930317Z outputs = self.rel_attn( 2025-11-03T16:41:13.9930655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:13.9931042Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:13.9931194Z 2025-11-03T16:41:13.9931289Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9931640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9931934Z res = mod(**inputs) 2025-11-03T16:41:13.9932273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9932650Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9933021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9933393Z outputs = layer_module( 2025-11-03T16:41:13.9933730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9934087Z outputs = self.rel_attn( 2025-11-03T16:41:13.9934442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9934801Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9935171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:13.9935601Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:13.9935782Z 2025-11-03T16:41:13.9935881Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9936213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9936511Z res = mod(**inputs) 2025-11-03T16:41:13.9936839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9937210Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9937578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9937934Z outputs = layer_module( 2025-11-03T16:41:13.9938281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9938648Z outputs = self.rel_attn( 2025-11-03T16:41:13.9938995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:13.9939412Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:13.9939588Z 2025-11-03T16:41:13.9939692Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9940028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9940319Z res = mod(**inputs) 2025-11-03T16:41:13.9940651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9941022Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9941405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9941757Z outputs = layer_module( 2025-11-03T16:41:13.9942104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9942462Z outputs = self.rel_attn( 2025-11-03T16:41:13.9942805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9943167Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9943529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:13.9943963Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:13.9944139Z 2025-11-03T16:41:13.9944237Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9944587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9944888Z res = mod(**inputs) 2025-11-03T16:41:13.9945224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9945605Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9945981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9946356Z outputs = layer_module( 2025-11-03T16:41:13.9946696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9947059Z outputs = self.rel_attn( 2025-11-03T16:41:13.9947408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:13.9947799Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:13.9947947Z 2025-11-03T16:41:13.9948054Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9948383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9948681Z res = mod(**inputs) 2025-11-03T16:41:13.9949016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9949398Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9949769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9950117Z outputs = layer_module( 2025-11-03T16:41:13.9950460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9950818Z outputs = self.rel_attn( 2025-11-03T16:41:13.9951168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9951523Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9951913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:13.9952338Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:13.9952502Z 2025-11-03T16:41:13.9952608Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9952941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9953233Z res = mod(**inputs) 2025-11-03T16:41:13.9953572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9953965Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9954468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9954852Z outputs = layer_module( 2025-11-03T16:41:13.9955221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9955606Z outputs = self.rel_attn( 2025-11-03T16:41:13.9955976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9956382Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9956835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9957278Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9957452Z 2025-11-03T16:41:13.9957556Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9957945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9958266Z res = mod(**inputs) 2025-11-03T16:41:13.9958613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9959007Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9959396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9959793Z outputs = layer_module( 2025-11-03T16:41:13.9960156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9960525Z outputs = self.rel_attn( 2025-11-03T16:41:13.9960889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:13.9961282Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:13.9961695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:13.9962124Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:13.9962295Z 2025-11-03T16:41:13.9962400Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9962755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9963070Z res = mod(**inputs) 2025-11-03T16:41:13.9963424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9963809Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9964196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9964571Z outputs = layer_module( 2025-11-03T16:41:13.9964919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:13.9965411Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:13.9965914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:13.9966290Z return forward_fn(*input_tensors) 2025-11-03T16:41:13.9966693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:13.9967068Z output_x = self.ff(output_x) 2025-11-03T16:41:13.9967433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:13.9967800Z output = self.layer_1(output) 2025-11-03T16:41:13.9967926Z 2025-11-03T16:41:13.9970906Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9971275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9971593Z res = mod(**inputs) 2025-11-03T16:41:13.9971941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9972323Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9972710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9973083Z outputs = layer_module( 2025-11-03T16:41:13.9973442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:13.9973935Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:13.9974501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:13.9974895Z return forward_fn(*input_tensors) 2025-11-03T16:41:13.9975269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:13.9975645Z output_x = self.ff(output_x) 2025-11-03T16:41:13.9976015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:13.9976425Z output = self.activation_function(output) 2025-11-03T16:41:13.9976778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:13.9977107Z return self.act(input) 2025-11-03T16:41:13.9977224Z 2025-11-03T16:41:13.9977324Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9977674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9977984Z res = mod(**inputs) 2025-11-03T16:41:13.9978324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9978713Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9979094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9979462Z outputs = layer_module( 2025-11-03T16:41:13.9979819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:13.9980318Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:13.9980844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:13.9981227Z return forward_fn(*input_tensors) 2025-11-03T16:41:13.9981611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:13.9982013Z output_x = self.ff(output_x) 2025-11-03T16:41:13.9982382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:13.9982767Z output = self.layer_2(output) 2025-11-03T16:41:13.9982902Z 2025-11-03T16:41:13.9983007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9983365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9983684Z res = mod(**inputs) 2025-11-03T16:41:13.9984030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9984434Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9984888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9985270Z outputs = layer_module( 2025-11-03T16:41:13.9985631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9986010Z outputs = self.rel_attn( 2025-11-03T16:41:13.9986382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:13.9986795Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:13.9986947Z 2025-11-03T16:41:13.9987058Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9987404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9987723Z res = mod(**inputs) 2025-11-03T16:41:13.9988096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9988491Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9988879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9989246Z outputs = layer_module( 2025-11-03T16:41:13.9989611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9990039Z outputs = self.rel_attn( 2025-11-03T16:41:13.9990403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:13.9990802Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:13.9990963Z 2025-11-03T16:41:13.9991064Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9991419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9991736Z res = mod(**inputs) 2025-11-03T16:41:13.9992115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9992485Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9992857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9993217Z outputs = layer_module( 2025-11-03T16:41:13.9993562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9993936Z outputs = self.rel_attn( 2025-11-03T16:41:13.9994376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:13.9994762Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:13.9995148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:13.9995591Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:13.9995796Z 2025-11-03T16:41:13.9995896Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:13.9996232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:13.9996532Z res = mod(**inputs) 2025-11-03T16:41:13.9996872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:13.9997251Z transformer_outputs = self.transformer( 2025-11-03T16:41:13.9997616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:13.9997976Z outputs = layer_module( 2025-11-03T16:41:13.9998356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:13.9998723Z outputs = self.rel_attn( 2025-11-03T16:41:13.9999065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:13.9999475Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:13.9999656Z 2025-11-03T16:41:13.9999753Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0000088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0000384Z res = mod(**inputs) 2025-11-03T16:41:14.0000712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0001088Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0001474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0001833Z outputs = layer_module( 2025-11-03T16:41:14.0002183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0002538Z outputs = self.rel_attn( 2025-11-03T16:41:14.0002887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0003249Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0003648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0004078Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0004252Z 2025-11-03T16:41:14.0004352Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0004693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0004996Z res = mod(**inputs) 2025-11-03T16:41:14.0005335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0005708Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0006083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0006448Z outputs = layer_module( 2025-11-03T16:41:14.0006798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0007160Z outputs = self.rel_attn( 2025-11-03T16:41:14.0007500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0007890Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0008041Z 2025-11-03T16:41:14.0008141Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0008484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0008800Z res = mod(**inputs) 2025-11-03T16:41:14.0009129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0009504Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0009874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0010235Z outputs = layer_module( 2025-11-03T16:41:14.0010574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0010931Z outputs = self.rel_attn( 2025-11-03T16:41:14.0011309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0011679Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0012051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0012473Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0012653Z 2025-11-03T16:41:14.0012751Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0013083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0013566Z res = mod(**inputs) 2025-11-03T16:41:14.0013917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0014290Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0014715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0015084Z outputs = layer_module( 2025-11-03T16:41:14.0015435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0015792Z outputs = self.rel_attn( 2025-11-03T16:41:14.0016141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0016522Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0016943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0017360Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0017516Z 2025-11-03T16:41:14.0017613Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0017952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0018256Z res = mod(**inputs) 2025-11-03T16:41:14.0018593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0018969Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0019331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0019687Z outputs = layer_module( 2025-11-03T16:41:14.0020031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0020390Z outputs = self.rel_attn( 2025-11-03T16:41:14.0020728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0021105Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0021501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0021917Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0022100Z 2025-11-03T16:41:14.0022204Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0022537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0022842Z res = mod(**inputs) 2025-11-03T16:41:14.0023183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0023568Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0023944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0024299Z outputs = layer_module( 2025-11-03T16:41:14.0024681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0025178Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0025681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0026059Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0026413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0026781Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0027134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0027496Z output = self.layer_1(output) 2025-11-03T16:41:14.0027611Z 2025-11-03T16:41:14.0027723Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0028072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0028374Z res = mod(**inputs) 2025-11-03T16:41:14.0028708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0029080Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0029451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0029816Z outputs = layer_module( 2025-11-03T16:41:14.0030161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0030649Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0031145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0031518Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0031873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0032238Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0032592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0032968Z output = self.activation_function(output) 2025-11-03T16:41:14.0033301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0033626Z return self.act(input) 2025-11-03T16:41:14.0033737Z 2025-11-03T16:41:14.0033833Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0034229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0034579Z res = mod(**inputs) 2025-11-03T16:41:14.0034934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0035358Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0035799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0036166Z outputs = layer_module( 2025-11-03T16:41:14.0036512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0037009Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0037520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0037907Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0038303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0038670Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0039039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0039412Z output = self.layer_2(output) 2025-11-03T16:41:14.0039532Z 2025-11-03T16:41:14.0039640Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0039987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0040292Z res = mod(**inputs) 2025-11-03T16:41:14.0040641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0041023Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0041423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0041793Z outputs = layer_module( 2025-11-03T16:41:14.0042139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0042511Z outputs = self.rel_attn( 2025-11-03T16:41:14.0042866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0043260Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0043424Z 2025-11-03T16:41:14.0043523Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0043865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0044176Z res = mod(**inputs) 2025-11-03T16:41:14.0044526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0044914Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0045285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0045663Z outputs = layer_module( 2025-11-03T16:41:14.0046014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0046383Z outputs = self.rel_attn( 2025-11-03T16:41:14.0046740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0047138Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0047293Z 2025-11-03T16:41:14.0047393Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0047737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0048043Z res = mod(**inputs) 2025-11-03T16:41:14.0048381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0048791Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0049167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0049526Z outputs = layer_module( 2025-11-03T16:41:14.0049869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0050221Z outputs = self.rel_attn( 2025-11-03T16:41:14.0050567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0050930Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0051308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0051753Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0051933Z 2025-11-03T16:41:14.0052033Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0052364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0052662Z res = mod(**inputs) 2025-11-03T16:41:14.0052998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0053365Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0053735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0054089Z outputs = layer_module( 2025-11-03T16:41:14.0054429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0054807Z outputs = self.rel_attn( 2025-11-03T16:41:14.0055145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0055569Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0055750Z 2025-11-03T16:41:14.0055848Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0056188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0056504Z res = mod(**inputs) 2025-11-03T16:41:14.0056832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0057206Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0057576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0057933Z outputs = layer_module( 2025-11-03T16:41:14.0058268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0058626Z outputs = self.rel_attn( 2025-11-03T16:41:14.0058971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0059333Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0059705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0060128Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0060307Z 2025-11-03T16:41:14.0060406Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0060738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0061036Z res = mod(**inputs) 2025-11-03T16:41:14.0061368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0061768Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0062132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0062488Z outputs = layer_module( 2025-11-03T16:41:14.0062832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0063188Z outputs = self.rel_attn( 2025-11-03T16:41:14.0063533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0063926Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0064074Z 2025-11-03T16:41:14.0064181Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0064543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0064844Z res = mod(**inputs) 2025-11-03T16:41:14.0065191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0065571Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0065951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0066324Z outputs = layer_module( 2025-11-03T16:41:14.0066668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0067031Z outputs = self.rel_attn( 2025-11-03T16:41:14.0067386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0067774Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0068158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0068605Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0068785Z 2025-11-03T16:41:14.0068888Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0069240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0069563Z res = mod(**inputs) 2025-11-03T16:41:14.0069901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0070284Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0070665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0071032Z outputs = layer_module( 2025-11-03T16:41:14.0071381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0071753Z outputs = self.rel_attn( 2025-11-03T16:41:14.0072108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0072499Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0072900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0073328Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0073495Z 2025-11-03T16:41:14.0073595Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0073947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0074351Z res = mod(**inputs) 2025-11-03T16:41:14.0074718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0075132Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0075545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0075929Z outputs = layer_module( 2025-11-03T16:41:14.0076299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0076680Z outputs = self.rel_attn( 2025-11-03T16:41:14.0077036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0077435Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0077859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0078319Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0078487Z 2025-11-03T16:41:14.0078594Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0078943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0079255Z res = mod(**inputs) 2025-11-03T16:41:14.0079605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0079994Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0080372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0080743Z outputs = layer_module( 2025-11-03T16:41:14.0081102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0081629Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0082141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0082523Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0082901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0083300Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0083671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0084039Z output = self.layer_1(output) 2025-11-03T16:41:14.0084162Z 2025-11-03T16:41:14.0084263Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0084612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0084926Z res = mod(**inputs) 2025-11-03T16:41:14.0085274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0085660Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0086048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0086418Z outputs = layer_module( 2025-11-03T16:41:14.0086785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0087321Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0087837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0088245Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0088637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0089037Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0089407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0089795Z output = self.activation_function(output) 2025-11-03T16:41:14.0090152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0090494Z return self.act(input) 2025-11-03T16:41:14.0090604Z 2025-11-03T16:41:14.0090712Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0091059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0091374Z res = mod(**inputs) 2025-11-03T16:41:14.0091747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0092147Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0092540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0092908Z outputs = layer_module( 2025-11-03T16:41:14.0093272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0093791Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0094282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0094664Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0095036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0095400Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0095750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0096109Z output = self.layer_2(output) 2025-11-03T16:41:14.0096225Z 2025-11-03T16:41:14.0096329Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0096657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0096971Z res = mod(**inputs) 2025-11-03T16:41:14.0097309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0097683Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0098044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0098401Z outputs = layer_module( 2025-11-03T16:41:14.0098746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0099106Z outputs = self.rel_attn( 2025-11-03T16:41:14.0099447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0099820Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0099971Z 2025-11-03T16:41:14.0100068Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0100402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0100700Z res = mod(**inputs) 2025-11-03T16:41:14.0101025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0101395Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0101762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0102138Z outputs = layer_module( 2025-11-03T16:41:14.0102482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0102843Z outputs = self.rel_attn( 2025-11-03T16:41:14.0103188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0103581Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0103731Z 2025-11-03T16:41:14.0103836Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0104177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0104470Z res = mod(**inputs) 2025-11-03T16:41:14.0104822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0105208Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0105581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0105935Z outputs = layer_module( 2025-11-03T16:41:14.0106284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0106648Z outputs = self.rel_attn( 2025-11-03T16:41:14.0107002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0107369Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0107738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0108190Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0108373Z 2025-11-03T16:41:14.0108469Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0108802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0109099Z res = mod(**inputs) 2025-11-03T16:41:14.0109427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0109799Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0110221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0110585Z outputs = layer_module( 2025-11-03T16:41:14.0110925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0111286Z outputs = self.rel_attn( 2025-11-03T16:41:14.0111637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0112055Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0112234Z 2025-11-03T16:41:14.0112338Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0112664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0112992Z res = mod(**inputs) 2025-11-03T16:41:14.0113388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0113476Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0113719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0113792Z outputs = layer_module( 2025-11-03T16:41:14.0114038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0114159Z outputs = self.rel_attn( 2025-11-03T16:41:14.0114456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0114531Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0114822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0114957Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0114961Z 2025-11-03T16:41:14.0115075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0115286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0115359Z res = mod(**inputs) 2025-11-03T16:41:14.0115660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0115744Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0116013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0116082Z outputs = layer_module( 2025-11-03T16:41:14.0116350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0116422Z outputs = self.rel_attn( 2025-11-03T16:41:14.0116685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0116796Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0116800Z 2025-11-03T16:41:14.0116907Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0117143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0117209Z res = mod(**inputs) 2025-11-03T16:41:14.0117472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0117560Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0117813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0117890Z outputs = layer_module( 2025-11-03T16:41:14.0118174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0118251Z outputs = self.rel_attn( 2025-11-03T16:41:14.0118506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0118579Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0118865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0118992Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0118997Z 2025-11-03T16:41:14.0119107Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0119306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0119380Z res = mod(**inputs) 2025-11-03T16:41:14.0119643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0119727Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0119989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0120056Z outputs = layer_module( 2025-11-03T16:41:14.0120320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0120389Z outputs = self.rel_attn( 2025-11-03T16:41:14.0120659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0120757Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0121034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0121159Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0121163Z 2025-11-03T16:41:14.0121266Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0121470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0121533Z res = mod(**inputs) 2025-11-03T16:41:14.0121809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0121904Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0122163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0122241Z outputs = layer_module( 2025-11-03T16:41:14.0122504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0122573Z outputs = self.rel_attn( 2025-11-03T16:41:14.0122837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0122927Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0123211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0123343Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0123348Z 2025-11-03T16:41:14.0123453Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0123663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0123730Z res = mod(**inputs) 2025-11-03T16:41:14.0123994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0124080Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0124359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0124426Z outputs = layer_module( 2025-11-03T16:41:14.0124679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0124901Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0125180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0125264Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0125509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0125580Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0125824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0125895Z output = self.layer_1(output) 2025-11-03T16:41:14.0125899Z 2025-11-03T16:41:14.0126007Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0126193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0126262Z res = mod(**inputs) 2025-11-03T16:41:14.0126504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0126600Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0126849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0126913Z outputs = layer_module( 2025-11-03T16:41:14.0127162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0127359Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0127610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0127691Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0127952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0128032Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0128280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0128372Z output = self.activation_function(output) 2025-11-03T16:41:14.0128581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0128649Z return self.act(input) 2025-11-03T16:41:14.0128653Z 2025-11-03T16:41:14.0128758Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0128947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0129016Z res = mod(**inputs) 2025-11-03T16:41:14.0129262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0129354Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0129602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0129668Z outputs = layer_module( 2025-11-03T16:41:14.0129917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0130112Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0130384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0130460Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0130702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0130781Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0131023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0131100Z output = self.layer_2(output) 2025-11-03T16:41:14.0131103Z 2025-11-03T16:41:14.0131200Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0131390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0131458Z res = mod(**inputs) 2025-11-03T16:41:14.0131699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0131788Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0132033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0132097Z outputs = layer_module( 2025-11-03T16:41:14.0132343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0132411Z outputs = self.rel_attn( 2025-11-03T16:41:14.0132678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0132769Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0132773Z 2025-11-03T16:41:14.0132881Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0133072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0133132Z res = mod(**inputs) 2025-11-03T16:41:14.0133379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0133457Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0133720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0133786Z outputs = layer_module( 2025-11-03T16:41:14.0134028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0134100Z outputs = self.rel_attn( 2025-11-03T16:41:14.0134340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0134443Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0134447Z 2025-11-03T16:41:14.0134544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0134743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0134804Z res = mod(**inputs) 2025-11-03T16:41:14.0135050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0135161Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0135404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0135476Z outputs = layer_module( 2025-11-03T16:41:14.0135718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0135784Z outputs = self.rel_attn( 2025-11-03T16:41:14.0136053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0136123Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0136392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0136521Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0136525Z 2025-11-03T16:41:14.0136634Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0136821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0136884Z res = mod(**inputs) 2025-11-03T16:41:14.0137141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0137220Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0137477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0137543Z outputs = layer_module( 2025-11-03T16:41:14.0137788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0137862Z outputs = self.rel_attn( 2025-11-03T16:41:14.0138110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0138243Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0138267Z 2025-11-03T16:41:14.0138366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0138560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0138621Z res = mod(**inputs) 2025-11-03T16:41:14.0138868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0138958Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0139201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0139274Z outputs = layer_module( 2025-11-03T16:41:14.0139532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0139600Z outputs = self.rel_attn( 2025-11-03T16:41:14.0139853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0139924Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0140196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0140321Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0140326Z 2025-11-03T16:41:14.0140424Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0140621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0140682Z res = mod(**inputs) 2025-11-03T16:41:14.0140940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0141035Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0141284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0141350Z outputs = layer_module( 2025-11-03T16:41:14.0141588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0141662Z outputs = self.rel_attn( 2025-11-03T16:41:14.0141914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0142016Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0142019Z 2025-11-03T16:41:14.0142117Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0142304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0142377Z res = mod(**inputs) 2025-11-03T16:41:14.0142626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0142714Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0142958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0143030Z outputs = layer_module( 2025-11-03T16:41:14.0143273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0143340Z outputs = self.rel_attn( 2025-11-03T16:41:14.0143590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0143657Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0143923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0144041Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0144060Z 2025-11-03T16:41:14.0144159Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0144354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0144415Z res = mod(**inputs) 2025-11-03T16:41:14.0144665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0144747Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0144987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0145059Z outputs = layer_module( 2025-11-03T16:41:14.0145303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0145393Z outputs = self.rel_attn( 2025-11-03T16:41:14.0145638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0145735Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0145992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0146100Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0146104Z 2025-11-03T16:41:14.0146210Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0146399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0146466Z res = mod(**inputs) 2025-11-03T16:41:14.0146708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0146803Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0147054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0147120Z outputs = layer_module( 2025-11-03T16:41:14.0147365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0147430Z outputs = self.rel_attn( 2025-11-03T16:41:14.0147677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0147779Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0148040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0148154Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0148158Z 2025-11-03T16:41:14.0148258Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0148454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0148517Z res = mod(**inputs) 2025-11-03T16:41:14.0148760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0148849Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0149090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0149163Z outputs = layer_module( 2025-11-03T16:41:14.0149405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0149612Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0149866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0149942Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0150212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0150283Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0150533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0150606Z output = self.layer_1(output) 2025-11-03T16:41:14.0150609Z 2025-11-03T16:41:14.0150710Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0150911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0150973Z res = mod(**inputs) 2025-11-03T16:41:14.0151269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0151351Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0151615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0151684Z outputs = layer_module( 2025-11-03T16:41:14.0151936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0152149Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0152408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0152492Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0152787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0152862Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0153119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0153207Z output = self.activation_function(output) 2025-11-03T16:41:14.0153428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0153496Z return self.act(input) 2025-11-03T16:41:14.0153499Z 2025-11-03T16:41:14.0153623Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0153823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0153885Z res = mod(**inputs) 2025-11-03T16:41:14.0154219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0154312Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0154586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0154657Z outputs = layer_module( 2025-11-03T16:41:14.0154923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0155151Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0155407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0155494Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0155746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0155818Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0156082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0156153Z output = self.layer_2(output) 2025-11-03T16:41:14.0156179Z 2025-11-03T16:41:14.0156289Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0156481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0156551Z res = mod(**inputs) 2025-11-03T16:41:14.0156804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0156888Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0157144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0157209Z outputs = layer_module( 2025-11-03T16:41:14.0157469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0157563Z outputs = self.rel_attn( 2025-11-03T16:41:14.0157817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0157921Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0157925Z 2025-11-03T16:41:14.0158025Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0158228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0158293Z res = mod(**inputs) 2025-11-03T16:41:14.0158557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0158637Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0158893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0158984Z outputs = layer_module( 2025-11-03T16:41:14.0159232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0159310Z outputs = self.rel_attn( 2025-11-03T16:41:14.0159555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0159655Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0159659Z 2025-11-03T16:41:14.0159783Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0159972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0160042Z res = mod(**inputs) 2025-11-03T16:41:14.0160291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0160379Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0160628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0160696Z outputs = layer_module( 2025-11-03T16:41:14.0160950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0161017Z outputs = self.rel_attn( 2025-11-03T16:41:14.0161268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0161341Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0161602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0161737Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0161740Z 2025-11-03T16:41:14.0161843Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0162040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0162102Z res = mod(**inputs) 2025-11-03T16:41:14.0162369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0162457Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0162707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0162780Z outputs = layer_module( 2025-11-03T16:41:14.0163030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0163103Z outputs = self.rel_attn( 2025-11-03T16:41:14.0163347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0163491Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0163495Z 2025-11-03T16:41:14.0163605Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0163802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0163870Z res = mod(**inputs) 2025-11-03T16:41:14.0164119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0164203Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0164462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0164525Z outputs = layer_module( 2025-11-03T16:41:14.0164777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0164855Z outputs = self.rel_attn( 2025-11-03T16:41:14.0165103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0165175Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0165432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0165562Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0165566Z 2025-11-03T16:41:14.0165680Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0165875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0165936Z res = mod(**inputs) 2025-11-03T16:41:14.0166180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0166269Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0166512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0166586Z outputs = layer_module( 2025-11-03T16:41:14.0166827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0166893Z outputs = self.rel_attn( 2025-11-03T16:41:14.0167144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0167241Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0167244Z 2025-11-03T16:41:14.0167352Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0167538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0167608Z res = mod(**inputs) 2025-11-03T16:41:14.0167852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0167933Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0168201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0168265Z outputs = layer_module( 2025-11-03T16:41:14.0168515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0168582Z outputs = self.rel_attn( 2025-11-03T16:41:14.0168823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0168901Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0169161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0169305Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0169309Z 2025-11-03T16:41:14.0169408Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0169606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0169667Z res = mod(**inputs) 2025-11-03T16:41:14.0169910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0169999Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0170240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0170312Z outputs = layer_module( 2025-11-03T16:41:14.0170565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0170631Z outputs = self.rel_attn( 2025-11-03T16:41:14.0170899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0170987Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0171255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0171363Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0171367Z 2025-11-03T16:41:14.0171490Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0171679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0171739Z res = mod(**inputs) 2025-11-03T16:41:14.0171988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0172067Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0172316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0172382Z outputs = layer_module( 2025-11-03T16:41:14.0172621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0172694Z outputs = self.rel_attn( 2025-11-03T16:41:14.0172934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0173029Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0173291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0173399Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0173410Z 2025-11-03T16:41:14.0173510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0173704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0173776Z res = mod(**inputs) 2025-11-03T16:41:14.0174043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0174130Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0174375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0174444Z outputs = layer_module( 2025-11-03T16:41:14.0174697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0174900Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0175178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0175256Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0175511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0175584Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0175829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0175909Z output = self.layer_1(output) 2025-11-03T16:41:14.0175914Z 2025-11-03T16:41:14.0176014Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0176213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0176276Z res = mod(**inputs) 2025-11-03T16:41:14.0176524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0176625Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0176875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0176950Z outputs = layer_module( 2025-11-03T16:41:14.0177204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0177407Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0177687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0177763Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0178020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0178091Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0178354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0178442Z output = self.activation_function(output) 2025-11-03T16:41:14.0178658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0178734Z return self.act(input) 2025-11-03T16:41:14.0178737Z 2025-11-03T16:41:14.0178837Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0179042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0179103Z res = mod(**inputs) 2025-11-03T16:41:14.0179353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0179440Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0179695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0179768Z outputs = layer_module( 2025-11-03T16:41:14.0180046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0180255Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0180513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0180589Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0180846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0180915Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0181172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0181257Z output = self.layer_2(output) 2025-11-03T16:41:14.0181261Z 2025-11-03T16:41:14.0181364Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0181572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0181633Z res = mod(**inputs) 2025-11-03T16:41:14.0181888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0181971Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0182226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0182292Z outputs = layer_module( 2025-11-03T16:41:14.0182544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0182633Z outputs = self.rel_attn( 2025-11-03T16:41:14.0182887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0182996Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0182999Z 2025-11-03T16:41:14.0183101Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0183293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0183363Z res = mod(**inputs) 2025-11-03T16:41:14.0183631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0183721Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0183971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0184036Z outputs = layer_module( 2025-11-03T16:41:14.0184299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0184367Z outputs = self.rel_attn( 2025-11-03T16:41:14.0184624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0184723Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0184726Z 2025-11-03T16:41:14.0184835Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0185029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0185090Z res = mod(**inputs) 2025-11-03T16:41:14.0185349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0185429Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0185689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0185754Z outputs = layer_module( 2025-11-03T16:41:14.0186021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0186096Z outputs = self.rel_attn( 2025-11-03T16:41:14.0186342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0186424Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0186691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0186828Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0186832Z 2025-11-03T16:41:14.0186933Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0187143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0187216Z res = mod(**inputs) 2025-11-03T16:41:14.0187468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0187558Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0187804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0187869Z outputs = layer_module( 2025-11-03T16:41:14.0188129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0188196Z outputs = self.rel_attn( 2025-11-03T16:41:14.0188451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0188599Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0188605Z 2025-11-03T16:41:14.0188715Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0188910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0188973Z res = mod(**inputs) 2025-11-03T16:41:14.0189234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0189316Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0189592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0189657Z outputs = layer_module( 2025-11-03T16:41:14.0189908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0189982Z outputs = self.rel_attn( 2025-11-03T16:41:14.0190233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0190311Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0190579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0190705Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0190716Z 2025-11-03T16:41:14.0190816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0191011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0191081Z res = mod(**inputs) 2025-11-03T16:41:14.0191332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0191421Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0191672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0191736Z outputs = layer_module( 2025-11-03T16:41:14.0192005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0192073Z outputs = self.rel_attn( 2025-11-03T16:41:14.0192331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0192426Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0192429Z 2025-11-03T16:41:14.0192526Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0192720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0192781Z res = mod(**inputs) 2025-11-03T16:41:14.0193034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0193126Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0193373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0193437Z outputs = layer_module( 2025-11-03T16:41:14.0193676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0193750Z outputs = self.rel_attn( 2025-11-03T16:41:14.0193990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0194066Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0194393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0194516Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0194541Z 2025-11-03T16:41:14.0194652Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0194854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0194927Z res = mod(**inputs) 2025-11-03T16:41:14.0195182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0195273Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0195546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0195612Z outputs = layer_module( 2025-11-03T16:41:14.0195864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0195929Z outputs = self.rel_attn( 2025-11-03T16:41:14.0196181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0196268Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0196532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0196652Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0196655Z 2025-11-03T16:41:14.0196760Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0196962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0197025Z res = mod(**inputs) 2025-11-03T16:41:14.0197269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0197357Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0197604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0197678Z outputs = layer_module( 2025-11-03T16:41:14.0197921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0198014Z outputs = self.rel_attn( 2025-11-03T16:41:14.0198256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0198340Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0198611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0198717Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0198721Z 2025-11-03T16:41:14.0198827Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0199034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0199098Z res = mod(**inputs) 2025-11-03T16:41:14.0199357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0199439Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0199693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0199757Z outputs = layer_module( 2025-11-03T16:41:14.0200010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0200210Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0200466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0200564Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0200813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0200892Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0201141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0201210Z output = self.layer_1(output) 2025-11-03T16:41:14.0201220Z 2025-11-03T16:41:14.0201344Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0201531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0201602Z res = mod(**inputs) 2025-11-03T16:41:14.0201848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0201935Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0202182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0202247Z outputs = layer_module( 2025-11-03T16:41:14.0202499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0202695Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0202953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0203029Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0203272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0203349Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0203592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0203685Z output = self.activation_function(output) 2025-11-03T16:41:14.0203910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0203984Z return self.act(input) 2025-11-03T16:41:14.0203987Z 2025-11-03T16:41:14.0204088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0204278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0204351Z res = mod(**inputs) 2025-11-03T16:41:14.0204595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0204679Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0204921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0205002Z outputs = layer_module( 2025-11-03T16:41:14.0205257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0205454Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0205713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0205787Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0206037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0206107Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0206349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0206425Z output = self.layer_2(output) 2025-11-03T16:41:14.0206445Z 2025-11-03T16:41:14.0206546Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0206743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0206806Z res = mod(**inputs) 2025-11-03T16:41:14.0207051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0207137Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0207397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0207469Z outputs = layer_module( 2025-11-03T16:41:14.0207708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0207774Z outputs = self.rel_attn( 2025-11-03T16:41:14.0208023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0208119Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0208123Z 2025-11-03T16:41:14.0208227Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0208416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0208484Z res = mod(**inputs) 2025-11-03T16:41:14.0208736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0208814Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0209056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0209119Z outputs = layer_module( 2025-11-03T16:41:14.0209363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0209429Z outputs = self.rel_attn( 2025-11-03T16:41:14.0209663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0209782Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0209786Z 2025-11-03T16:41:14.0209884Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0210079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0210143Z res = mod(**inputs) 2025-11-03T16:41:14.0210402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0210479Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0210715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0210803Z outputs = layer_module( 2025-11-03T16:41:14.0211045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0211121Z outputs = self.rel_attn( 2025-11-03T16:41:14.0211365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0211435Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0211701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0211830Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0211834Z 2025-11-03T16:41:14.0211940Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0212129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0212212Z res = mod(**inputs) 2025-11-03T16:41:14.0212461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0212546Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0212815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0212878Z outputs = layer_module( 2025-11-03T16:41:14.0213122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0213355Z outputs = self.rel_attn( 2025-11-03T16:41:14.0213603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0213739Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0213742Z 2025-11-03T16:41:14.0213844Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0214036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0214098Z res = mod(**inputs) 2025-11-03T16:41:14.0214342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0214432Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0214696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0214772Z outputs = layer_module( 2025-11-03T16:41:14.0215015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0215090Z outputs = self.rel_attn( 2025-11-03T16:41:14.0215334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0215406Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0215679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0215841Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0215844Z 2025-11-03T16:41:14.0215946Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0216124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0216185Z res = mod(**inputs) 2025-11-03T16:41:14.0216429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0216506Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0216748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0216836Z outputs = layer_module( 2025-11-03T16:41:14.0217080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0217147Z outputs = self.rel_attn( 2025-11-03T16:41:14.0217383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0217485Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0217489Z 2025-11-03T16:41:14.0217585Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0217777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0217838Z res = mod(**inputs) 2025-11-03T16:41:14.0218085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0218196Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0218436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0218507Z outputs = layer_module( 2025-11-03T16:41:14.0218741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0218804Z outputs = self.rel_attn( 2025-11-03T16:41:14.0219045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0219136Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0219394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0219507Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0219511Z 2025-11-03T16:41:14.0219615Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0219796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0219856Z res = mod(**inputs) 2025-11-03T16:41:14.0220098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0220175Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0220413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0220476Z outputs = layer_module( 2025-11-03T16:41:14.0220715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0220786Z outputs = self.rel_attn( 2025-11-03T16:41:14.0221032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0221123Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0221376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0221508Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0221512Z 2025-11-03T16:41:14.0221607Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0221789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0221857Z res = mod(**inputs) 2025-11-03T16:41:14.0222093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0222177Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0222410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0222487Z outputs = layer_module( 2025-11-03T16:41:14.0222731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0222799Z outputs = self.rel_attn( 2025-11-03T16:41:14.0223042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0223123Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0223382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0223488Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0223491Z 2025-11-03T16:41:14.0223585Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0223775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0223865Z res = mod(**inputs) 2025-11-03T16:41:14.0224112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0224190Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0224426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0224496Z outputs = layer_module( 2025-11-03T16:41:14.0224730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0224947Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0225193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0225274Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0225518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0225586Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0225831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0225899Z output = self.layer_1(output) 2025-11-03T16:41:14.0225903Z 2025-11-03T16:41:14.0226006Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0226193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0226253Z res = mod(**inputs) 2025-11-03T16:41:14.0226498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0226575Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0226820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0226883Z outputs = layer_module( 2025-11-03T16:41:14.0227122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0227656Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0227911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0227998Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0228248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0228330Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0228580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0228682Z output = self.activation_function(output) 2025-11-03T16:41:14.0228904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0228976Z return self.act(input) 2025-11-03T16:41:14.0228979Z 2025-11-03T16:41:14.0229086Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0229278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0229341Z res = mod(**inputs) 2025-11-03T16:41:14.0229592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0229671Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0229923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0229988Z outputs = layer_module( 2025-11-03T16:41:14.0230250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0230450Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0230700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0230782Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0231042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0231118Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0231360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0231428Z output = self.layer_2(output) 2025-11-03T16:41:14.0231439Z 2025-11-03T16:41:14.0231540Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0231732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0231805Z res = mod(**inputs) 2025-11-03T16:41:14.0232049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0232135Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0232378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0232444Z outputs = layer_module( 2025-11-03T16:41:14.0232693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0232762Z outputs = self.rel_attn( 2025-11-03T16:41:14.0233014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0233112Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0233115Z 2025-11-03T16:41:14.0233215Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0233435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0233497Z res = mod(**inputs) 2025-11-03T16:41:14.0233747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0233829Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0234082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0234207Z outputs = layer_module( 2025-11-03T16:41:14.0234461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0234541Z outputs = self.rel_attn( 2025-11-03T16:41:14.0234810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0234917Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0234921Z 2025-11-03T16:41:14.0235022Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0235223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0235297Z res = mod(**inputs) 2025-11-03T16:41:14.0235538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0235631Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0235893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0235960Z outputs = layer_module( 2025-11-03T16:41:14.0236224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0236292Z outputs = self.rel_attn( 2025-11-03T16:41:14.0236549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0236622Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0236890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0237035Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0237038Z 2025-11-03T16:41:14.0237140Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0237347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0237410Z res = mod(**inputs) 2025-11-03T16:41:14.0237664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0237744Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0237986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0238058Z outputs = layer_module( 2025-11-03T16:41:14.0238302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0238377Z outputs = self.rel_attn( 2025-11-03T16:41:14.0238623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0238756Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0238759Z 2025-11-03T16:41:14.0238856Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0239047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0239119Z res = mod(**inputs) 2025-11-03T16:41:14.0239380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0239469Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0239722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0239787Z outputs = layer_module( 2025-11-03T16:41:14.0240039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0240104Z outputs = self.rel_attn( 2025-11-03T16:41:14.0240353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0240422Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0240703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0240828Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0240832Z 2025-11-03T16:41:14.0240930Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0241126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0241187Z res = mod(**inputs) 2025-11-03T16:41:14.0241442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0241519Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0241767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0241835Z outputs = layer_module( 2025-11-03T16:41:14.0242092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0242167Z outputs = self.rel_attn( 2025-11-03T16:41:14.0242406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0242500Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0242511Z 2025-11-03T16:41:14.0242609Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0242811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0242882Z res = mod(**inputs) 2025-11-03T16:41:14.0243121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0243208Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0243451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0243512Z outputs = layer_module( 2025-11-03T16:41:14.0243758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0243822Z outputs = self.rel_attn( 2025-11-03T16:41:14.0244071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0244145Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0244400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0244526Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0244529Z 2025-11-03T16:41:14.0244628Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0244825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0244885Z res = mod(**inputs) 2025-11-03T16:41:14.0245132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0245228Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0245471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0245544Z outputs = layer_module( 2025-11-03T16:41:14.0245785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0245860Z outputs = self.rel_attn( 2025-11-03T16:41:14.0246103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0246188Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0246515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0246624Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0246628Z 2025-11-03T16:41:14.0246734Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0246922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0246989Z res = mod(**inputs) 2025-11-03T16:41:14.0247235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0247314Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0247566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0247630Z outputs = layer_module( 2025-11-03T16:41:14.0247893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0247961Z outputs = self.rel_attn( 2025-11-03T16:41:14.0248205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0248296Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0248558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0248688Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0248691Z 2025-11-03T16:41:14.0248790Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0248985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0249045Z res = mod(**inputs) 2025-11-03T16:41:14.0249291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0249379Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0249637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0249712Z outputs = layer_module( 2025-11-03T16:41:14.0249950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0250145Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0250398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0250469Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0250712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0250780Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0251017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0251106Z output = self.layer_1(output) 2025-11-03T16:41:14.0251110Z 2025-11-03T16:41:14.0251204Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0251398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0251459Z res = mod(**inputs) 2025-11-03T16:41:14.0251702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0251779Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0252016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0252101Z outputs = layer_module( 2025-11-03T16:41:14.0252335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0252534Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0252777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0252851Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0253103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0253172Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0253419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0253504Z output = self.activation_function(output) 2025-11-03T16:41:14.0253729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0253800Z return self.act(input) 2025-11-03T16:41:14.0253804Z 2025-11-03T16:41:14.0253902Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0254099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0254160Z res = mod(**inputs) 2025-11-03T16:41:14.0254412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0254509Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0254756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0254830Z outputs = layer_module( 2025-11-03T16:41:14.0255079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0255285Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0255541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0255625Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0255872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0255944Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0256202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0256273Z output = self.layer_2(output) 2025-11-03T16:41:14.0256276Z 2025-11-03T16:41:14.0256383Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0256576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0256636Z res = mod(**inputs) 2025-11-03T16:41:14.0256904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0256983Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0257232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0257298Z outputs = layer_module( 2025-11-03T16:41:14.0257546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0257614Z outputs = self.rel_attn( 2025-11-03T16:41:14.0257854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0257956Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0257972Z 2025-11-03T16:41:14.0258072Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0258269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0258332Z res = mod(**inputs) 2025-11-03T16:41:14.0258573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0258662Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0258902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0258973Z outputs = layer_module( 2025-11-03T16:41:14.0259210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0259277Z outputs = self.rel_attn( 2025-11-03T16:41:14.0259543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0259639Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0259644Z 2025-11-03T16:41:14.0259749Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0259934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0260002Z res = mod(**inputs) 2025-11-03T16:41:14.0260245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0260339Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0260588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0260655Z outputs = layer_module( 2025-11-03T16:41:14.0260905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0260971Z outputs = self.rel_attn( 2025-11-03T16:41:14.0261211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0261291Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0261551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0261685Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0261688Z 2025-11-03T16:41:14.0261785Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0261981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0262040Z res = mod(**inputs) 2025-11-03T16:41:14.0262285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0262371Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0262613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0262701Z outputs = layer_module( 2025-11-03T16:41:14.0262941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0263008Z outputs = self.rel_attn( 2025-11-03T16:41:14.0263256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0263381Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0263384Z 2025-11-03T16:41:14.0263487Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0263674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0263759Z res = mod(**inputs) 2025-11-03T16:41:14.0264006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0264086Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0264338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0264403Z outputs = layer_module( 2025-11-03T16:41:14.0264654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0264718Z outputs = self.rel_attn( 2025-11-03T16:41:14.0264959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0265035Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0265308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0265440Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0265445Z 2025-11-03T16:41:14.0265544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0265738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0265799Z res = mod(**inputs) 2025-11-03T16:41:14.0266047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0266147Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0266388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0266458Z outputs = layer_module( 2025-11-03T16:41:14.0266707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0266772Z outputs = self.rel_attn( 2025-11-03T16:41:14.0267022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0267119Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0267122Z 2025-11-03T16:41:14.0267227Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0267414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0267476Z res = mod(**inputs) 2025-11-03T16:41:14.0267732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0267811Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0268060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0268128Z outputs = layer_module( 2025-11-03T16:41:14.0268381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0268475Z outputs = self.rel_attn( 2025-11-03T16:41:14.0268716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0268791Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0269050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0269175Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0269179Z 2025-11-03T16:41:14.0269276Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0269463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0269547Z res = mod(**inputs) 2025-11-03T16:41:14.0269796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0269886Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0270134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0270217Z outputs = layer_module( 2025-11-03T16:41:14.0270468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0270535Z outputs = self.rel_attn( 2025-11-03T16:41:14.0270793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0270879Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0271161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0271270Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0271275Z 2025-11-03T16:41:14.0271374Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0271569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0271632Z res = mod(**inputs) 2025-11-03T16:41:14.0271886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0271982Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0272221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0272293Z outputs = layer_module( 2025-11-03T16:41:14.0272538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0272608Z outputs = self.rel_attn( 2025-11-03T16:41:14.0272848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0272943Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0273201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0273307Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0273311Z 2025-11-03T16:41:14.0273418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0273606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0273675Z res = mod(**inputs) 2025-11-03T16:41:14.0273919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0273999Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0274318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0274416Z outputs = layer_module( 2025-11-03T16:41:14.0274682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0274887Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0275161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0275240Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0275492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0275574Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0275851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0275932Z output = self.layer_1(output) 2025-11-03T16:41:14.0275935Z 2025-11-03T16:41:14.0276036Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0276228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0276299Z res = mod(**inputs) 2025-11-03T16:41:14.0276547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0276634Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0276876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0276950Z outputs = layer_module( 2025-11-03T16:41:14.0277218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0277418Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0277675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0277747Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0278001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0278087Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0278327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0278421Z output = self.activation_function(output) 2025-11-03T16:41:14.0278632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0278709Z return self.act(input) 2025-11-03T16:41:14.0278712Z 2025-11-03T16:41:14.0278811Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0279008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0279069Z res = mod(**inputs) 2025-11-03T16:41:14.0279310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0279400Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0279646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0279719Z outputs = layer_module( 2025-11-03T16:41:14.0279958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0280158Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0280412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0280502Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0280752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0280823Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0281074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0281142Z output = self.layer_2(output) 2025-11-03T16:41:14.0281146Z 2025-11-03T16:41:14.0281245Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0281443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0281519Z res = mod(**inputs) 2025-11-03T16:41:14.0281775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0281856Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0282102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0282174Z outputs = layer_module( 2025-11-03T16:41:14.0282416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0282489Z outputs = self.rel_attn( 2025-11-03T16:41:14.0282731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0282833Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0282845Z 2025-11-03T16:41:14.0282965Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0283156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0283228Z res = mod(**inputs) 2025-11-03T16:41:14.0283471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0283558Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0283802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0283881Z outputs = layer_module( 2025-11-03T16:41:14.0284128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0284196Z outputs = self.rel_attn( 2025-11-03T16:41:14.0284449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0284546Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0284549Z 2025-11-03T16:41:14.0284651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0284848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0284910Z res = mod(**inputs) 2025-11-03T16:41:14.0285161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0285243Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0285496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0285559Z outputs = layer_module( 2025-11-03T16:41:14.0285801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0285877Z outputs = self.rel_attn( 2025-11-03T16:41:14.0286116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0286211Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0286465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0286589Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0286594Z 2025-11-03T16:41:14.0286701Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0286889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0286956Z res = mod(**inputs) 2025-11-03T16:41:14.0287206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0287310Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0287562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0287627Z outputs = layer_module( 2025-11-03T16:41:14.0287874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0287939Z outputs = self.rel_attn( 2025-11-03T16:41:14.0288180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0288305Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0288308Z 2025-11-03T16:41:14.0288404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0288597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0288658Z res = mod(**inputs) 2025-11-03T16:41:14.0288923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0289003Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0289240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0289311Z outputs = layer_module( 2025-11-03T16:41:14.0289548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0289646Z outputs = self.rel_attn( 2025-11-03T16:41:14.0289884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0289960Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0290219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0290347Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0290351Z 2025-11-03T16:41:14.0290458Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0290648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0290715Z res = mod(**inputs) 2025-11-03T16:41:14.0290966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0291046Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0291289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0291352Z outputs = layer_module( 2025-11-03T16:41:14.0291594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0291659Z outputs = self.rel_attn( 2025-11-03T16:41:14.0291906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0292014Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0292017Z 2025-11-03T16:41:14.0292112Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0292301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0292363Z res = mod(**inputs) 2025-11-03T16:41:14.0292606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0292682Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0292913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0292983Z outputs = layer_module( 2025-11-03T16:41:14.0293230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0293303Z outputs = self.rel_attn( 2025-11-03T16:41:14.0293541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0293608Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0293866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0293983Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0293986Z 2025-11-03T16:41:14.0294088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0294270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0294338Z res = mod(**inputs) 2025-11-03T16:41:14.0294594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0294673Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0294919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0294982Z outputs = layer_module( 2025-11-03T16:41:14.0295225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0295302Z outputs = self.rel_attn( 2025-11-03T16:41:14.0295538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0295629Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0295883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0295999Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0296002Z 2025-11-03T16:41:14.0296097Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0296289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0296350Z res = mod(**inputs) 2025-11-03T16:41:14.0296589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0296674Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0296910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0296980Z outputs = layer_module( 2025-11-03T16:41:14.0297218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0297283Z outputs = self.rel_attn( 2025-11-03T16:41:14.0297532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0297632Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0297895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0298000Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0298005Z 2025-11-03T16:41:14.0298106Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0298291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0298350Z res = mod(**inputs) 2025-11-03T16:41:14.0298600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0298682Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0298943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0299009Z outputs = layer_module( 2025-11-03T16:41:14.0299246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0299449Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0299695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0299774Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0300012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0300088Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0300338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0300407Z output = self.layer_1(output) 2025-11-03T16:41:14.0300411Z 2025-11-03T16:41:14.0300515Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0300701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0300767Z res = mod(**inputs) 2025-11-03T16:41:14.0301001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0301093Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0301339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0301402Z outputs = layer_module( 2025-11-03T16:41:14.0301648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0301843Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0302099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0302172Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0302412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0302489Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0302729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0302822Z output = self.activation_function(output) 2025-11-03T16:41:14.0303026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0303095Z return self.act(input) 2025-11-03T16:41:14.0303099Z 2025-11-03T16:41:14.0303205Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0303420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0303486Z res = mod(**inputs) 2025-11-03T16:41:14.0303719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0303795Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0304039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0304102Z outputs = layer_module( 2025-11-03T16:41:14.0304346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0304552Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0304802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0304874Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0305110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0305184Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0305423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0305500Z output = self.layer_2(output) 2025-11-03T16:41:14.0305504Z 2025-11-03T16:41:14.0305601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0305789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0305856Z res = mod(**inputs) 2025-11-03T16:41:14.0306110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0306199Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0306443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0306512Z outputs = layer_module( 2025-11-03T16:41:14.0306757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0306840Z outputs = self.rel_attn( 2025-11-03T16:41:14.0307089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0307182Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0307185Z 2025-11-03T16:41:14.0307292Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0307482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0307544Z res = mod(**inputs) 2025-11-03T16:41:14.0307795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0307874Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0308123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0308189Z outputs = layer_module( 2025-11-03T16:41:14.0308438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0308502Z outputs = self.rel_attn( 2025-11-03T16:41:14.0308744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0308849Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0308852Z 2025-11-03T16:41:14.0308949Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0309161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0309222Z res = mod(**inputs) 2025-11-03T16:41:14.0309468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0309555Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0309798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0309868Z outputs = layer_module( 2025-11-03T16:41:14.0310108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0310172Z outputs = self.rel_attn( 2025-11-03T16:41:14.0310443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0310515Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0310781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0310907Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0310910Z 2025-11-03T16:41:14.0311015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0311208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0311269Z res = mod(**inputs) 2025-11-03T16:41:14.0311521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0311601Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0311877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0311942Z outputs = layer_module( 2025-11-03T16:41:14.0312187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0312261Z outputs = self.rel_attn( 2025-11-03T16:41:14.0312498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0312648Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0312651Z 2025-11-03T16:41:14.0312748Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0312947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0313008Z res = mod(**inputs) 2025-11-03T16:41:14.0313451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0313548Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0313791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0313866Z outputs = layer_module( 2025-11-03T16:41:14.0314160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0314240Z outputs = self.rel_attn( 2025-11-03T16:41:14.0314505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0314579Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0314854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0314983Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0314987Z 2025-11-03T16:41:14.0315106Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0315341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0315405Z res = mod(**inputs) 2025-11-03T16:41:14.0315665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0315750Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0316013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0316082Z outputs = layer_module( 2025-11-03T16:41:14.0316329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0316405Z outputs = self.rel_attn( 2025-11-03T16:41:14.0316687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0316792Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0316798Z 2025-11-03T16:41:14.0316898Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0317090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0317152Z res = mod(**inputs) 2025-11-03T16:41:14.0317396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0317486Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0317734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0317806Z outputs = layer_module( 2025-11-03T16:41:14.0318070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0318137Z outputs = self.rel_attn( 2025-11-03T16:41:14.0318392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0318461Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0318724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0318868Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0318871Z 2025-11-03T16:41:14.0318969Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0319164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0319226Z res = mod(**inputs) 2025-11-03T16:41:14.0319482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0319562Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0319818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0319883Z outputs = layer_module( 2025-11-03T16:41:14.0320127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0320198Z outputs = self.rel_attn( 2025-11-03T16:41:14.0320443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0320533Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0320796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0320906Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0320910Z 2025-11-03T16:41:14.0321015Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0321203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0321289Z res = mod(**inputs) 2025-11-03T16:41:14.0321535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0321620Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0321864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0321927Z outputs = layer_module( 2025-11-03T16:41:14.0322177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0322242Z outputs = self.rel_attn( 2025-11-03T16:41:14.0322510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0322596Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0322855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0322969Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0322972Z 2025-11-03T16:41:14.0323069Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0323265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0323327Z res = mod(**inputs) 2025-11-03T16:41:14.0323578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0323663Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0323910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0323982Z outputs = layer_module( 2025-11-03T16:41:14.0324218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0324421Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0324667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0324754Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0325002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0325070Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0325314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0325382Z output = self.layer_1(output) 2025-11-03T16:41:14.0325385Z 2025-11-03T16:41:14.0325488Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0325678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0325737Z res = mod(**inputs) 2025-11-03T16:41:14.0325980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0326057Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0326298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0326364Z outputs = layer_module( 2025-11-03T16:41:14.0326599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0326800Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0327044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0327143Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0327380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0327447Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0327700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0327782Z output = self.activation_function(output) 2025-11-03T16:41:14.0327991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0328055Z return self.act(input) 2025-11-03T16:41:14.0328058Z 2025-11-03T16:41:14.0328178Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0328362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0328423Z res = mod(**inputs) 2025-11-03T16:41:14.0328667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0328743Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0328984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0329049Z outputs = layer_module( 2025-11-03T16:41:14.0329282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0329481Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0329740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0329820Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0330054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0330128Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0330366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0330448Z output = self.layer_2(output) 2025-11-03T16:41:14.0330452Z 2025-11-03T16:41:14.0330555Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0330740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0330806Z res = mod(**inputs) 2025-11-03T16:41:14.0331046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0331122Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0331368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0331430Z outputs = layer_module( 2025-11-03T16:41:14.0331676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0331743Z outputs = self.rel_attn( 2025-11-03T16:41:14.0331990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0332084Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0332087Z 2025-11-03T16:41:14.0332182Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0332378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0332442Z res = mod(**inputs) 2025-11-03T16:41:14.0332694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0332787Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0333036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0333106Z outputs = layer_module( 2025-11-03T16:41:14.0333341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0333413Z outputs = self.rel_attn( 2025-11-03T16:41:14.0333647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0333740Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0333751Z 2025-11-03T16:41:14.0333869Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0334053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0334122Z res = mod(**inputs) 2025-11-03T16:41:14.0334362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0334445Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0334680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0334745Z outputs = layer_module( 2025-11-03T16:41:14.0334985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0335048Z outputs = self.rel_attn( 2025-11-03T16:41:14.0335312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0335381Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0335634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0335767Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0335771Z 2025-11-03T16:41:14.0335868Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0336058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0336133Z res = mod(**inputs) 2025-11-03T16:41:14.0336384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0336461Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0336710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0336779Z outputs = layer_module( 2025-11-03T16:41:14.0337022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0337094Z outputs = self.rel_attn( 2025-11-03T16:41:14.0337341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0337465Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0337470Z 2025-11-03T16:41:14.0337574Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0337761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0337827Z res = mod(**inputs) 2025-11-03T16:41:14.0338073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0338160Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0338403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0338513Z outputs = layer_module( 2025-11-03T16:41:14.0338759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0338824Z outputs = self.rel_attn( 2025-11-03T16:41:14.0339066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0339135Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0339389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0339518Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0339521Z 2025-11-03T16:41:14.0339635Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0339827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0339888Z res = mod(**inputs) 2025-11-03T16:41:14.0340126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0340211Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0340448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0340519Z outputs = layer_module( 2025-11-03T16:41:14.0340756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0340827Z outputs = self.rel_attn( 2025-11-03T16:41:14.0341080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0341173Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0341177Z 2025-11-03T16:41:14.0341281Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0341465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0341533Z res = mod(**inputs) 2025-11-03T16:41:14.0341767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0341860Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0342104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0342167Z outputs = layer_module( 2025-11-03T16:41:14.0342410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0342477Z outputs = self.rel_attn( 2025-11-03T16:41:14.0342719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0342787Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0343037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0343158Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0343163Z 2025-11-03T16:41:14.0343258Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0343450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0343510Z res = mod(**inputs) 2025-11-03T16:41:14.0343748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0343838Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0344075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0344161Z outputs = layer_module( 2025-11-03T16:41:14.0344403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0344468Z outputs = self.rel_attn( 2025-11-03T16:41:14.0344715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0344798Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0345070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0345176Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0345180Z 2025-11-03T16:41:14.0345305Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0345493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0345557Z res = mod(**inputs) 2025-11-03T16:41:14.0345810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0345889Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0346142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0346209Z outputs = layer_module( 2025-11-03T16:41:14.0346457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0346532Z outputs = self.rel_attn( 2025-11-03T16:41:14.0346810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0346902Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0347158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0347273Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0347277Z 2025-11-03T16:41:14.0347375Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0347564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0347649Z res = mod(**inputs) 2025-11-03T16:41:14.0347893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0347980Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0348221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0348286Z outputs = layer_module( 2025-11-03T16:41:14.0348536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0348737Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0348995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0349069Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0349318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0349386Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0349634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0349713Z output = self.layer_1(output) 2025-11-03T16:41:14.0349717Z 2025-11-03T16:41:14.0349816Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0350011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0350089Z res = mod(**inputs) 2025-11-03T16:41:14.0350332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0350418Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0350663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0350734Z outputs = layer_module( 2025-11-03T16:41:14.0350973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0351194Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0351451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0351527Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0351779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0351848Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0352096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0352182Z output = self.activation_function(output) 2025-11-03T16:41:14.0352389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0352463Z return self.act(input) 2025-11-03T16:41:14.0352467Z 2025-11-03T16:41:14.0352564Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0352775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0352837Z res = mod(**inputs) 2025-11-03T16:41:14.0353088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0353168Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0353411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0353502Z outputs = layer_module( 2025-11-03T16:41:14.0353744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0353949Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0354266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0354345Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0354606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0354678Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0354935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0355006Z output = self.layer_2(output) 2025-11-03T16:41:14.0355011Z 2025-11-03T16:41:14.0355125Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0355321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0355385Z res = mod(**inputs) 2025-11-03T16:41:14.0355646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0355730Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0355995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0356096Z outputs = layer_module( 2025-11-03T16:41:14.0356342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0356423Z outputs = self.rel_attn( 2025-11-03T16:41:14.0356674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0356781Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0356785Z 2025-11-03T16:41:14.0356887Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0357088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0357158Z res = mod(**inputs) 2025-11-03T16:41:14.0357413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0357504Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0357746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0357817Z outputs = layer_module( 2025-11-03T16:41:14.0358056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0358123Z outputs = self.rel_attn( 2025-11-03T16:41:14.0358372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0358468Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0358471Z 2025-11-03T16:41:14.0358577Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0358779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0358843Z res = mod(**inputs) 2025-11-03T16:41:14.0359093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0359171Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0359422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0359505Z outputs = layer_module( 2025-11-03T16:41:14.0359753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0359819Z outputs = self.rel_attn( 2025-11-03T16:41:14.0360056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0360135Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0360389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0360522Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0360528Z 2025-11-03T16:41:14.0360626Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0360812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0360887Z res = mod(**inputs) 2025-11-03T16:41:14.0361129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0361216Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0361457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0361522Z outputs = layer_module( 2025-11-03T16:41:14.0361772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0361860Z outputs = self.rel_attn( 2025-11-03T16:41:14.0362111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0362235Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0362238Z 2025-11-03T16:41:14.0362344Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0362533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0362592Z res = mod(**inputs) 2025-11-03T16:41:14.0362845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0362923Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0363190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0363256Z outputs = layer_module( 2025-11-03T16:41:14.0363497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0363569Z outputs = self.rel_attn( 2025-11-03T16:41:14.0363822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0363902Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0364160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0364290Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0364293Z 2025-11-03T16:41:14.0364393Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0364601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0364670Z res = mod(**inputs) 2025-11-03T16:41:14.0364913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0365001Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0365247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0365329Z outputs = layer_module( 2025-11-03T16:41:14.0365583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0365648Z outputs = self.rel_attn( 2025-11-03T16:41:14.0365904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0366002Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0366006Z 2025-11-03T16:41:14.0366112Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0366305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0366365Z res = mod(**inputs) 2025-11-03T16:41:14.0366622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0366703Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0366959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0367022Z outputs = layer_module( 2025-11-03T16:41:14.0367272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0367346Z outputs = self.rel_attn( 2025-11-03T16:41:14.0367594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0367687Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0367948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0368067Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0368078Z 2025-11-03T16:41:14.0368178Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0368371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0368439Z res = mod(**inputs) 2025-11-03T16:41:14.0368687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0368774Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0369035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0369103Z outputs = layer_module( 2025-11-03T16:41:14.0369359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0369425Z outputs = self.rel_attn( 2025-11-03T16:41:14.0369677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0369763Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0370032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0370151Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0370155Z 2025-11-03T16:41:14.0370255Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0370472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0370537Z res = mod(**inputs) 2025-11-03T16:41:14.0370796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0370879Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0371143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0371235Z outputs = layer_module( 2025-11-03T16:41:14.0371482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0371559Z outputs = self.rel_attn( 2025-11-03T16:41:14.0371807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0371893Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0372178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0372290Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0372293Z 2025-11-03T16:41:14.0372399Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0372597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0372667Z res = mod(**inputs) 2025-11-03T16:41:14.0372930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0373013Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0373280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0373349Z outputs = layer_module( 2025-11-03T16:41:14.0373616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0373845Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0374117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0374201Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0374469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0374550Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0374807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0374888Z output = self.layer_1(output) 2025-11-03T16:41:14.0374891Z 2025-11-03T16:41:14.0375012Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0375218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0375291Z res = mod(**inputs) 2025-11-03T16:41:14.0375553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0375646Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0375903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0375971Z outputs = layer_module( 2025-11-03T16:41:14.0376241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0376450Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0376735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0376814Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0377085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0377157Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0377422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0377543Z output = self.activation_function(output) 2025-11-03T16:41:14.0377781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0377858Z return self.act(input) 2025-11-03T16:41:14.0377861Z 2025-11-03T16:41:14.0377965Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0378170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0378241Z res = mod(**inputs) 2025-11-03T16:41:14.0378523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0378617Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0378895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0378964Z outputs = layer_module( 2025-11-03T16:41:14.0379234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0379445Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0379722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0379801Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0380070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0380159Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0380427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0380507Z output = self.layer_2(output) 2025-11-03T16:41:14.0380511Z 2025-11-03T16:41:14.0380616Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0380823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0380887Z res = mod(**inputs) 2025-11-03T16:41:14.0381153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0381246Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0381528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0381606Z outputs = layer_module( 2025-11-03T16:41:14.0381882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0381971Z outputs = self.rel_attn( 2025-11-03T16:41:14.0382216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0382312Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0382315Z 2025-11-03T16:41:14.0382421Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0382612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0382682Z res = mod(**inputs) 2025-11-03T16:41:14.0382950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0383031Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0383292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0383360Z outputs = layer_module( 2025-11-03T16:41:14.0383620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0383705Z outputs = self.rel_attn( 2025-11-03T16:41:14.0383960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0384063Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0384066Z 2025-11-03T16:41:14.0384163Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0384363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0384425Z res = mod(**inputs) 2025-11-03T16:41:14.0384679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0384760Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0385005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0385077Z outputs = layer_module( 2025-11-03T16:41:14.0385324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0385406Z outputs = self.rel_attn( 2025-11-03T16:41:14.0385650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0385721Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0385989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0386116Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0386135Z 2025-11-03T16:41:14.0386242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0386432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0386500Z res = mod(**inputs) 2025-11-03T16:41:14.0386750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0386832Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0387087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0387151Z outputs = layer_module( 2025-11-03T16:41:14.0387418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0387485Z outputs = self.rel_attn( 2025-11-03T16:41:14.0387729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0387863Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0387867Z 2025-11-03T16:41:14.0387965Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0388162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0388223Z res = mod(**inputs) 2025-11-03T16:41:14.0388478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0388557Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0388815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0388888Z outputs = layer_module( 2025-11-03T16:41:14.0389132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0389208Z outputs = self.rel_attn( 2025-11-03T16:41:14.0389454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0389523Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0389810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0389934Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0389938Z 2025-11-03T16:41:14.0390043Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0390234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0390299Z res = mod(**inputs) 2025-11-03T16:41:14.0390544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0390627Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0390879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0390943Z outputs = layer_module( 2025-11-03T16:41:14.0391195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0391260Z outputs = self.rel_attn( 2025-11-03T16:41:14.0391501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0391603Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0391608Z 2025-11-03T16:41:14.0391709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0391905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0391980Z res = mod(**inputs) 2025-11-03T16:41:14.0392229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0392313Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0411266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0411545Z outputs = layer_module( 2025-11-03T16:41:14.0411872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0411945Z outputs = self.rel_attn( 2025-11-03T16:41:14.0412320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0412401Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0412665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0412809Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0412818Z 2025-11-03T16:41:14.0412932Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0413139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0413376Z res = mod(**inputs) 2025-11-03T16:41:14.0413643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0413742Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0414063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0414140Z outputs = layer_module( 2025-11-03T16:41:14.0414386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0414454Z outputs = self.rel_attn( 2025-11-03T16:41:14.0414699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0414790Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0415102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0415213Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0415217Z 2025-11-03T16:41:14.0415328Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0415523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0415590Z res = mod(**inputs) 2025-11-03T16:41:14.0415843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0415929Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0416180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0416246Z outputs = layer_module( 2025-11-03T16:41:14.0416487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0416562Z outputs = self.rel_attn( 2025-11-03T16:41:14.0416801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0416896Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0417160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0417275Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0417318Z 2025-11-03T16:41:14.0417418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0417610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0417682Z res = mod(**inputs) 2025-11-03T16:41:14.0417922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0418015Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0418255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0418320Z outputs = layer_module( 2025-11-03T16:41:14.0418605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0418806Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0419065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0419142Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0419386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0419458Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0419696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0419778Z output = self.layer_1(output) 2025-11-03T16:41:14.0419782Z 2025-11-03T16:41:14.0419884Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0420102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0420165Z res = mod(**inputs) 2025-11-03T16:41:14.0420406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0420497Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0420733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0420822Z outputs = layer_module( 2025-11-03T16:41:14.0421059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0421262Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0421510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0421586Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0421834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0421906Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0422153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0422239Z output = self.activation_function(output) 2025-11-03T16:41:14.0422444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0422521Z return self.act(input) 2025-11-03T16:41:14.0422524Z 2025-11-03T16:41:14.0422623Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0422816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0422879Z res = mod(**inputs) 2025-11-03T16:41:14.0423126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0423232Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0423470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0423541Z outputs = layer_module( 2025-11-03T16:41:14.0423774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0423976Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0424228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0424300Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0424563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0424631Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0424879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0424947Z output = self.layer_2(output) 2025-11-03T16:41:14.0424951Z 2025-11-03T16:41:14.0425057Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0425245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0425307Z res = mod(**inputs) 2025-11-03T16:41:14.0425552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0425630Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0425893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0425960Z outputs = layer_module( 2025-11-03T16:41:14.0426195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0426273Z outputs = self.rel_attn( 2025-11-03T16:41:14.0426510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0426616Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0426635Z 2025-11-03T16:41:14.0426732Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0426918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0426986Z res = mod(**inputs) 2025-11-03T16:41:14.0427224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0427312Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0427550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0427622Z outputs = layer_module( 2025-11-03T16:41:14.0427857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0427923Z outputs = self.rel_attn( 2025-11-03T16:41:14.0428168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0428266Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0428269Z 2025-11-03T16:41:14.0428373Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0428557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0428619Z res = mod(**inputs) 2025-11-03T16:41:14.0428867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0428962Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0429205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0429270Z outputs = layer_module( 2025-11-03T16:41:14.0429512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0429579Z outputs = self.rel_attn( 2025-11-03T16:41:14.0429819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0429898Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0430173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0430315Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0430321Z 2025-11-03T16:41:14.0430420Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0430609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0430680Z res = mod(**inputs) 2025-11-03T16:41:14.0430936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0431024Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0431271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0431336Z outputs = layer_module( 2025-11-03T16:41:14.0431602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0431673Z outputs = self.rel_attn( 2025-11-03T16:41:14.0431924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0432055Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0432059Z 2025-11-03T16:41:14.0432166Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0432353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0432433Z res = mod(**inputs) 2025-11-03T16:41:14.0432691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0432772Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0433032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0433102Z outputs = layer_module( 2025-11-03T16:41:14.0433349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0433425Z outputs = self.rel_attn( 2025-11-03T16:41:14.0433671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0433752Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0434023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0434238Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0434244Z 2025-11-03T16:41:14.0434351Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0434556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0434631Z res = mod(**inputs) 2025-11-03T16:41:14.0434892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0435016Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0435261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0435326Z outputs = layer_module( 2025-11-03T16:41:14.0435620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0435692Z outputs = self.rel_attn( 2025-11-03T16:41:14.0435943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0436039Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0436043Z 2025-11-03T16:41:14.0436150Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0436356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0436421Z res = mod(**inputs) 2025-11-03T16:41:14.0436677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0436758Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0437008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0437076Z outputs = layer_module( 2025-11-03T16:41:14.0437321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0437397Z outputs = self.rel_attn( 2025-11-03T16:41:14.0437642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0437745Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0438002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0438126Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0438137Z 2025-11-03T16:41:14.0438237Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0438427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0438520Z res = mod(**inputs) 2025-11-03T16:41:14.0438765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0438854Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0439100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0439166Z outputs = layer_module( 2025-11-03T16:41:14.0439417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0439487Z outputs = self.rel_attn( 2025-11-03T16:41:14.0439739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0439826Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0440092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0440210Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0440213Z 2025-11-03T16:41:14.0440313Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0440512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0440573Z res = mod(**inputs) 2025-11-03T16:41:14.0440831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0440939Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0441182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0441256Z outputs = layer_module( 2025-11-03T16:41:14.0441496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0441570Z outputs = self.rel_attn( 2025-11-03T16:41:14.0441810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0441897Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0442182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0442291Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0442295Z 2025-11-03T16:41:14.0442405Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0442594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0442664Z res = mod(**inputs) 2025-11-03T16:41:14.0442910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0442991Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0443241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0443308Z outputs = layer_module( 2025-11-03T16:41:14.0443558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0443775Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0444026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0444111Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0444354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0444450Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0444692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0444770Z output = self.layer_1(output) 2025-11-03T16:41:14.0444774Z 2025-11-03T16:41:14.0444871Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0445060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0445131Z res = mod(**inputs) 2025-11-03T16:41:14.0445373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0445460Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0445703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0445766Z outputs = layer_module( 2025-11-03T16:41:14.0446018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0446214Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0446470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0446545Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0446799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0446884Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0447126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0447221Z output = self.activation_function(output) 2025-11-03T16:41:14.0447431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0447511Z return self.act(input) 2025-11-03T16:41:14.0447514Z 2025-11-03T16:41:14.0447615Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0447801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0447871Z res = mod(**inputs) 2025-11-03T16:41:14.0448130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0448218Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0448462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0448539Z outputs = layer_module( 2025-11-03T16:41:14.0448779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0448970Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0449220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0449290Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0449578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0449645Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0449880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0449959Z output = self.layer_2(output) 2025-11-03T16:41:14.0449963Z 2025-11-03T16:41:14.0450062Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0450253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0450331Z res = mod(**inputs) 2025-11-03T16:41:14.0450578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0450665Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0450908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0450984Z outputs = layer_module( 2025-11-03T16:41:14.0451227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0451304Z outputs = self.rel_attn( 2025-11-03T16:41:14.0451550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0451643Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0451647Z 2025-11-03T16:41:14.0451753Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0451936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0452004Z res = mod(**inputs) 2025-11-03T16:41:14.0452241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0452317Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0452564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0452643Z outputs = layer_module( 2025-11-03T16:41:14.0452888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0452953Z outputs = self.rel_attn( 2025-11-03T16:41:14.0453193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0453299Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0453303Z 2025-11-03T16:41:14.0453402Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0453599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0453670Z res = mod(**inputs) 2025-11-03T16:41:14.0453950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0454030Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0454275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0454348Z outputs = layer_module( 2025-11-03T16:41:14.0454588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0454667Z outputs = self.rel_attn( 2025-11-03T16:41:14.0454920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0454989Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0455249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0455388Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0455392Z 2025-11-03T16:41:14.0455496Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0455682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0455751Z res = mod(**inputs) 2025-11-03T16:41:14.0455988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0456080Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0456325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0456388Z outputs = layer_module( 2025-11-03T16:41:14.0456632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0456695Z outputs = self.rel_attn( 2025-11-03T16:41:14.0456931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0457067Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0457070Z 2025-11-03T16:41:14.0457165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0457356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0457416Z res = mod(**inputs) 2025-11-03T16:41:14.0457665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0457743Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0457980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0458051Z outputs = layer_module( 2025-11-03T16:41:14.0458293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0458366Z outputs = self.rel_attn( 2025-11-03T16:41:14.0458617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0458685Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0458942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0459065Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0459069Z 2025-11-03T16:41:14.0459175Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0459355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0459422Z res = mod(**inputs) 2025-11-03T16:41:14.0459674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0459754Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0460002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0460067Z outputs = layer_module( 2025-11-03T16:41:14.0460310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0460375Z outputs = self.rel_attn( 2025-11-03T16:41:14.0460609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0460712Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0460716Z 2025-11-03T16:41:14.0460811Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0461014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0461077Z res = mod(**inputs) 2025-11-03T16:41:14.0461314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0461402Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0461649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0461743Z outputs = layer_module( 2025-11-03T16:41:14.0461991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0462068Z outputs = self.rel_attn( 2025-11-03T16:41:14.0462310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0462388Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0462648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0462772Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0462783Z 2025-11-03T16:41:14.0462883Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0463070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0463138Z res = mod(**inputs) 2025-11-03T16:41:14.0463385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0463471Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0463712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0463777Z outputs = layer_module( 2025-11-03T16:41:14.0464028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0464095Z outputs = self.rel_attn( 2025-11-03T16:41:14.0464366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0464453Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0464713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0464831Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0464834Z 2025-11-03T16:41:14.0464931Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0465127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0465187Z res = mod(**inputs) 2025-11-03T16:41:14.0465462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0465543Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0465790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0465863Z outputs = layer_module( 2025-11-03T16:41:14.0466105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0466182Z outputs = self.rel_attn( 2025-11-03T16:41:14.0466426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0466512Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0466782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0466907Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0466911Z 2025-11-03T16:41:14.0467018Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0467210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0467279Z res = mod(**inputs) 2025-11-03T16:41:14.0467522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0467601Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0467867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0467931Z outputs = layer_module( 2025-11-03T16:41:14.0468183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0468385Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0468640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0468725Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0468969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0469046Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0469288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0469372Z output = self.layer_1(output) 2025-11-03T16:41:14.0469375Z 2025-11-03T16:41:14.0469474Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0469671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0469732Z res = mod(**inputs) 2025-11-03T16:41:14.0469975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0470078Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0470316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0470388Z outputs = layer_module( 2025-11-03T16:41:14.0470628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0470826Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0471079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0471154Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0471421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0471494Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0471748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0471833Z output = self.activation_function(output) 2025-11-03T16:41:14.0472039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0472114Z return self.act(input) 2025-11-03T16:41:14.0472118Z 2025-11-03T16:41:14.0472217Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0472412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0472473Z res = mod(**inputs) 2025-11-03T16:41:14.0472735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0472821Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0473062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0473134Z outputs = layer_module( 2025-11-03T16:41:14.0473373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0473578Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0474073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0474225Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0474492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0474568Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0474823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0474897Z output = self.layer_2(output) 2025-11-03T16:41:14.0474901Z 2025-11-03T16:41:14.0475002Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0475203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0475265Z res = mod(**inputs) 2025-11-03T16:41:14.0475546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0475628Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0475883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0475952Z outputs = layer_module( 2025-11-03T16:41:14.0476204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0476281Z outputs = self.rel_attn( 2025-11-03T16:41:14.0476551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0476660Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0476664Z 2025-11-03T16:41:14.0476765Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0476961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0477031Z res = mod(**inputs) 2025-11-03T16:41:14.0477280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0477366Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0477634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0477702Z outputs = layer_module( 2025-11-03T16:41:14.0477957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0478024Z outputs = self.rel_attn( 2025-11-03T16:41:14.0478276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0478376Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0478380Z 2025-11-03T16:41:14.0478485Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0478678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0478738Z res = mod(**inputs) 2025-11-03T16:41:14.0479012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0479095Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0479353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0479419Z outputs = layer_module( 2025-11-03T16:41:14.0479667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0479739Z outputs = self.rel_attn( 2025-11-03T16:41:14.0480003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0480082Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0480346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0480481Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0480486Z 2025-11-03T16:41:14.0480589Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0480780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0480852Z res = mod(**inputs) 2025-11-03T16:41:14.0481103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0481189Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0481437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0481502Z outputs = layer_module( 2025-11-03T16:41:14.0481753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0481820Z outputs = self.rel_attn( 2025-11-03T16:41:14.0482077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0482211Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0482232Z 2025-11-03T16:41:14.0482338Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0482530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0482591Z res = mod(**inputs) 2025-11-03T16:41:14.0482850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0482935Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0483192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0483259Z outputs = layer_module( 2025-11-03T16:41:14.0483524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0483601Z outputs = self.rel_attn( 2025-11-03T16:41:14.0483854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0483933Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0484202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0484330Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0484342Z 2025-11-03T16:41:14.0484442Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0484635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0484704Z res = mod(**inputs) 2025-11-03T16:41:14.0484979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0485070Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0485318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0485385Z outputs = layer_module( 2025-11-03T16:41:14.0485638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0485705Z outputs = self.rel_attn( 2025-11-03T16:41:14.0485976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0486073Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0486077Z 2025-11-03T16:41:14.0486177Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0486376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0486441Z res = mod(**inputs) 2025-11-03T16:41:14.0486696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0486778Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0487035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0487102Z outputs = layer_module( 2025-11-03T16:41:14.0487347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0487423Z outputs = self.rel_attn( 2025-11-03T16:41:14.0487670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0487746Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0488011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0488132Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0488154Z 2025-11-03T16:41:14.0488261Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0488454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0488522Z res = mod(**inputs) 2025-11-03T16:41:14.0488773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0488860Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0489114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0489179Z outputs = layer_module( 2025-11-03T16:41:14.0489450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0489517Z outputs = self.rel_attn( 2025-11-03T16:41:14.0489769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0489857Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0490124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0490239Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0490244Z 2025-11-03T16:41:14.0490343Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0490539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0490602Z res = mod(**inputs) 2025-11-03T16:41:14.0490848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0490952Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0491206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0491277Z outputs = layer_module( 2025-11-03T16:41:14.0491515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0491584Z outputs = self.rel_attn( 2025-11-03T16:41:14.0491833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0491913Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0492172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0492278Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0492282Z 2025-11-03T16:41:14.0492385Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0492571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0492631Z res = mod(**inputs) 2025-11-03T16:41:14.0492874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0492949Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0493192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0493255Z outputs = layer_module( 2025-11-03T16:41:14.0493495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0493689Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0493933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0494026Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0494262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0494334Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0494566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0494634Z output = self.layer_1(output) 2025-11-03T16:41:14.0494644Z 2025-11-03T16:41:14.0494740Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0494921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0494987Z res = mod(**inputs) 2025-11-03T16:41:14.0495237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0495320Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0495559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0495622Z outputs = layer_module( 2025-11-03T16:41:14.0495864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0496055Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0496303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0496374Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0496629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0496704Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0496938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0497028Z output = self.activation_function(output) 2025-11-03T16:41:14.0497225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0497298Z return self.act(input) 2025-11-03T16:41:14.0497318Z 2025-11-03T16:41:14.0497415Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0497597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0497664Z res = mod(**inputs) 2025-11-03T16:41:14.0497898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0497983Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0498218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0498283Z outputs = layer_module( 2025-11-03T16:41:14.0498525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0498715Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0498963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0499033Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0499277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0499343Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0499576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0499650Z output = self.layer_2(output) 2025-11-03T16:41:14.0499668Z 2025-11-03T16:41:14.0499765Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0499953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0500011Z res = mod(**inputs) 2025-11-03T16:41:14.0500250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0500334Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0500569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0500639Z outputs = layer_module( 2025-11-03T16:41:14.0500887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0500952Z outputs = self.rel_attn( 2025-11-03T16:41:14.0501200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0501296Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0501299Z 2025-11-03T16:41:14.0501401Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0501582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0501648Z res = mod(**inputs) 2025-11-03T16:41:14.0501884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0501961Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0502220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0502286Z outputs = layer_module( 2025-11-03T16:41:14.0502523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0502590Z outputs = self.rel_attn( 2025-11-03T16:41:14.0502824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0502922Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0502943Z 2025-11-03T16:41:14.0503039Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0503226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0503287Z res = mod(**inputs) 2025-11-03T16:41:14.0503540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0503621Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0503864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0503937Z outputs = layer_module( 2025-11-03T16:41:14.0504173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0504248Z outputs = self.rel_attn( 2025-11-03T16:41:14.0504484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0504556Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0504815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0504939Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0504942Z 2025-11-03T16:41:14.0505047Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0505232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0505307Z res = mod(**inputs) 2025-11-03T16:41:14.0505552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0505628Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0505867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0505930Z outputs = layer_module( 2025-11-03T16:41:14.0506173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0506239Z outputs = self.rel_attn( 2025-11-03T16:41:14.0506478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0506631Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0506635Z 2025-11-03T16:41:14.0506734Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0506935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0506994Z res = mod(**inputs) 2025-11-03T16:41:14.0507230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0507315Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0507546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0507614Z outputs = layer_module( 2025-11-03T16:41:14.0507849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0507936Z outputs = self.rel_attn( 2025-11-03T16:41:14.0508176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0508247Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0508522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0508648Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0508674Z 2025-11-03T16:41:14.0508780Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0508970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0509031Z res = mod(**inputs) 2025-11-03T16:41:14.0509283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0509362Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0509617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0509684Z outputs = layer_module( 2025-11-03T16:41:14.0509932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0509997Z outputs = self.rel_attn( 2025-11-03T16:41:14.0510236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0510341Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0510344Z 2025-11-03T16:41:14.0510441Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0510638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0510698Z res = mod(**inputs) 2025-11-03T16:41:14.0510948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0511035Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0511298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0511371Z outputs = layer_module( 2025-11-03T16:41:14.0511613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0511681Z outputs = self.rel_attn( 2025-11-03T16:41:14.0511932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0512001Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0512265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0512398Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0512403Z 2025-11-03T16:41:14.0512508Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0512703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0512764Z res = mod(**inputs) 2025-11-03T16:41:14.0513022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0513103Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0513489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0513559Z outputs = layer_module( 2025-11-03T16:41:14.0513806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0513924Z outputs = self.rel_attn( 2025-11-03T16:41:14.0514236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0514345Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0514625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0514749Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0514754Z 2025-11-03T16:41:14.0514885Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0515091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0515161Z res = mod(**inputs) 2025-11-03T16:41:14.0515414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0515504Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0515807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0515885Z outputs = layer_module( 2025-11-03T16:41:14.0516133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0516198Z outputs = self.rel_attn( 2025-11-03T16:41:14.0516448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0516533Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0516806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0516913Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0516916Z 2025-11-03T16:41:14.0517019Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0517217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0517301Z res = mod(**inputs) 2025-11-03T16:41:14.0517556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0517635Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0517877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0517951Z outputs = layer_module( 2025-11-03T16:41:14.0518194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0518401Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0518678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0518760Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0519005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0519073Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0519321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0519391Z output = self.layer_1(output) 2025-11-03T16:41:14.0519394Z 2025-11-03T16:41:14.0519497Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0519684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0519743Z res = mod(**inputs) 2025-11-03T16:41:14.0519995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0520097Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0520352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0520417Z outputs = layer_module( 2025-11-03T16:41:14.0520658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0520864Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0521129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0521207Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0521453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0521531Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0521783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0521868Z output = self.activation_function(output) 2025-11-03T16:41:14.0522078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0522145Z return self.act(input) 2025-11-03T16:41:14.0522149Z 2025-11-03T16:41:14.0522257Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0522448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0522511Z res = mod(**inputs) 2025-11-03T16:41:14.0522763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0522839Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0523092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0523156Z outputs = layer_module( 2025-11-03T16:41:14.0523419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0523615Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0523864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0523944Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0524187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0524263Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0524519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0524601Z output = self.layer_2(output) 2025-11-03T16:41:14.0524611Z 2025-11-03T16:41:14.0524709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0524896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0524963Z res = mod(**inputs) 2025-11-03T16:41:14.0525201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0525286Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0525520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0525582Z outputs = layer_module( 2025-11-03T16:41:14.0525828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0525910Z outputs = self.rel_attn( 2025-11-03T16:41:14.0526152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0526247Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0526251Z 2025-11-03T16:41:14.0526345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0526538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0526618Z res = mod(**inputs) 2025-11-03T16:41:14.0526860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0526937Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0527188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0527254Z outputs = layer_module( 2025-11-03T16:41:14.0527491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0527565Z outputs = self.rel_attn( 2025-11-03T16:41:14.0527798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0527898Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0527901Z 2025-11-03T16:41:14.0527995Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0528180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0528246Z res = mod(**inputs) 2025-11-03T16:41:14.0528484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0528568Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0528805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0528869Z outputs = layer_module( 2025-11-03T16:41:14.0529135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0529200Z outputs = self.rel_attn( 2025-11-03T16:41:14.0529441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0529513Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0529773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0529896Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0529899Z 2025-11-03T16:41:14.0529994Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0530200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0530261Z res = mod(**inputs) 2025-11-03T16:41:14.0530505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0530586Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0530825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0530896Z outputs = layer_module( 2025-11-03T16:41:14.0531130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0531201Z outputs = self.rel_attn( 2025-11-03T16:41:14.0531434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0531578Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0531583Z 2025-11-03T16:41:14.0531679Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0531867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0531933Z res = mod(**inputs) 2025-11-03T16:41:14.0532172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0532256Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0532513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0532576Z outputs = layer_module( 2025-11-03T16:41:14.0532815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0532878Z outputs = self.rel_attn( 2025-11-03T16:41:14.0533121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0533189Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0533443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0533569Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0533572Z 2025-11-03T16:41:14.0533665Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0533855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0533913Z res = mod(**inputs) 2025-11-03T16:41:14.0534155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0534231Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0534469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0534540Z outputs = layer_module( 2025-11-03T16:41:14.0534789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0534860Z outputs = self.rel_attn( 2025-11-03T16:41:14.0535090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0535184Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0535194Z 2025-11-03T16:41:14.0535287Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0535468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0535535Z res = mod(**inputs) 2025-11-03T16:41:14.0535786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0535871Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0536104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0536167Z outputs = layer_module( 2025-11-03T16:41:14.0536406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0536471Z outputs = self.rel_attn( 2025-11-03T16:41:14.0536711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0536779Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0537027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0537150Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0537168Z 2025-11-03T16:41:14.0537264Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0537453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0537514Z res = mod(**inputs) 2025-11-03T16:41:14.0537755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0537831Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0538083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0538155Z outputs = layer_module( 2025-11-03T16:41:14.0538387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0538457Z outputs = self.rel_attn( 2025-11-03T16:41:14.0538692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0538775Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0539036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0539139Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0539142Z 2025-11-03T16:41:14.0539245Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0539428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0539493Z res = mod(**inputs) 2025-11-03T16:41:14.0539730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0539807Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0540052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0540115Z outputs = layer_module( 2025-11-03T16:41:14.0540370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0540435Z outputs = self.rel_attn( 2025-11-03T16:41:14.0540670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0540762Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0541014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0541127Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0541131Z 2025-11-03T16:41:14.0541225Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0541466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0541535Z res = mod(**inputs) 2025-11-03T16:41:14.0541768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0541853Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0542088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0542159Z outputs = layer_module( 2025-11-03T16:41:14.0542392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0542584Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0542832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0542922Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0543166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0543236Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0543470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0543544Z output = self.layer_1(output) 2025-11-03T16:41:14.0543547Z 2025-11-03T16:41:14.0543660Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0543849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0543908Z res = mod(**inputs) 2025-11-03T16:41:14.0544154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0544233Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0544478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0544551Z outputs = layer_module( 2025-11-03T16:41:14.0544786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0544984Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0545232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0545305Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0545557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0545625Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0545871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0545956Z output = self.activation_function(output) 2025-11-03T16:41:14.0546179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0546244Z return self.act(input) 2025-11-03T16:41:14.0546247Z 2025-11-03T16:41:14.0546342Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0546534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0546594Z res = mod(**inputs) 2025-11-03T16:41:14.0546840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0546918Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0547168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0547257Z outputs = layer_module( 2025-11-03T16:41:14.0547493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0547690Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0547930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0548009Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0548244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0548310Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0548557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0548638Z output = self.layer_2(output) 2025-11-03T16:41:14.0548642Z 2025-11-03T16:41:14.0548744Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0548928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0548986Z res = mod(**inputs) 2025-11-03T16:41:14.0549228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0549304Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0549570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0549633Z outputs = layer_module( 2025-11-03T16:41:14.0549865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0549936Z outputs = self.rel_attn( 2025-11-03T16:41:14.0550171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0550271Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0550276Z 2025-11-03T16:41:14.0550371Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0550563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0550623Z res = mod(**inputs) 2025-11-03T16:41:14.0550861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0550947Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0551180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0551249Z outputs = layer_module( 2025-11-03T16:41:14.0551483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0551548Z outputs = self.rel_attn( 2025-11-03T16:41:14.0551787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0551897Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0551900Z 2025-11-03T16:41:14.0552002Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0552184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0552250Z res = mod(**inputs) 2025-11-03T16:41:14.0552484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0552561Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0552805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0552882Z outputs = layer_module( 2025-11-03T16:41:14.0553126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0553191Z outputs = self.rel_attn( 2025-11-03T16:41:14.0553423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0553500Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0553753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0553888Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0553891Z 2025-11-03T16:41:14.0553991Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0554264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0554353Z res = mod(**inputs) 2025-11-03T16:41:14.0554614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0554706Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0554960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0555037Z outputs = layer_module( 2025-11-03T16:41:14.0555289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0555404Z outputs = self.rel_attn( 2025-11-03T16:41:14.0555650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0555772Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0555776Z 2025-11-03T16:41:14.0555888Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0556086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0556160Z res = mod(**inputs) 2025-11-03T16:41:14.0556417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0556499Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0556758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0556828Z outputs = layer_module( 2025-11-03T16:41:14.0557090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0557158Z outputs = self.rel_attn( 2025-11-03T16:41:14.0557410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0557490Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0557756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0557907Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0557911Z 2025-11-03T16:41:14.0558012Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0558205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0558275Z res = mod(**inputs) 2025-11-03T16:41:14.0558526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0558614Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0558865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0558954Z outputs = layer_module( 2025-11-03T16:41:14.0559206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0559274Z outputs = self.rel_attn( 2025-11-03T16:41:14.0559526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0559622Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0559626Z 2025-11-03T16:41:14.0559733Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0559922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0559984Z res = mod(**inputs) 2025-11-03T16:41:14.0560241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0560342Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0560599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0560667Z outputs = layer_module( 2025-11-03T16:41:14.0560919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0560986Z outputs = self.rel_attn( 2025-11-03T16:41:14.0561233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0561326Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0561593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0561720Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0561723Z 2025-11-03T16:41:14.0561824Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0562017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0562087Z res = mod(**inputs) 2025-11-03T16:41:14.0562342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0562430Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0562684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0562752Z outputs = layer_module( 2025-11-03T16:41:14.0563009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0563077Z outputs = self.rel_attn( 2025-11-03T16:41:14.0563334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0563423Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0563705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0563824Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0563828Z 2025-11-03T16:41:14.0563922Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0564110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0564171Z res = mod(**inputs) 2025-11-03T16:41:14.0564412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0564489Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0564721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0564805Z outputs = layer_module( 2025-11-03T16:41:14.0565038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0565109Z outputs = self.rel_attn( 2025-11-03T16:41:14.0565342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0565429Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0565688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0565791Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0565795Z 2025-11-03T16:41:14.0565898Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0566079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0566162Z res = mod(**inputs) 2025-11-03T16:41:14.0566399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0566478Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0566724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0566788Z outputs = layer_module( 2025-11-03T16:41:14.0567031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0567241Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0567488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0567561Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0567800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0567877Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0568112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0568188Z output = self.layer_1(output) 2025-11-03T16:41:14.0568191Z 2025-11-03T16:41:14.0568288Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0568475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0568545Z res = mod(**inputs) 2025-11-03T16:41:14.0568788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0568871Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0569122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0569194Z outputs = layer_module( 2025-11-03T16:41:14.0569451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0569649Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0569907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0569984Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0570234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0570302Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0570545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0570652Z output = self.activation_function(output) 2025-11-03T16:41:14.0570857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0570931Z return self.act(input) 2025-11-03T16:41:14.0570935Z 2025-11-03T16:41:14.0571034Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0571228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0571292Z res = mod(**inputs) 2025-11-03T16:41:14.0571536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0571622Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0571868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0571940Z outputs = layer_module( 2025-11-03T16:41:14.0572207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0572404Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0572661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0572734Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0573001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0573071Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0573309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0573387Z output = self.layer_2(output) 2025-11-03T16:41:14.0573391Z 2025-11-03T16:41:14.0573490Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0573687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0573750Z res = mod(**inputs) 2025-11-03T16:41:14.0573997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0574074Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0574313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0574387Z outputs = layer_module( 2025-11-03T16:41:14.0574625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0574697Z outputs = self.rel_attn( 2025-11-03T16:41:14.0574939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0575033Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0575044Z 2025-11-03T16:41:14.0575159Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0575346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0575414Z res = mod(**inputs) 2025-11-03T16:41:14.0575656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0575743Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0575986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0576049Z outputs = layer_module( 2025-11-03T16:41:14.0576292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0576375Z outputs = self.rel_attn( 2025-11-03T16:41:14.0576626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0576722Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0576726Z 2025-11-03T16:41:14.0576824Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0577019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0577082Z res = mod(**inputs) 2025-11-03T16:41:14.0577333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0577412Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0577661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0577740Z outputs = layer_module( 2025-11-03T16:41:14.0577982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0578059Z outputs = self.rel_attn( 2025-11-03T16:41:14.0578299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0578378Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0578637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0578779Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0578783Z 2025-11-03T16:41:14.0578890Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0579076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0579143Z res = mod(**inputs) 2025-11-03T16:41:14.0579392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0579472Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0579730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0579795Z outputs = layer_module( 2025-11-03T16:41:14.0580040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0580107Z outputs = self.rel_attn( 2025-11-03T16:41:14.0580353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0580479Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0580482Z 2025-11-03T16:41:14.0580579Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0580774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0580835Z res = mod(**inputs) 2025-11-03T16:41:14.0581107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0581186Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0581428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0581502Z outputs = layer_module( 2025-11-03T16:41:14.0581744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0581818Z outputs = self.rel_attn( 2025-11-03T16:41:14.0582058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0582138Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0582414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0582544Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0582548Z 2025-11-03T16:41:14.0582658Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0582852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0582924Z res = mod(**inputs) 2025-11-03T16:41:14.0583178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0583259Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0583536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0583617Z outputs = layer_module( 2025-11-03T16:41:14.0583874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0583941Z outputs = self.rel_attn( 2025-11-03T16:41:14.0584180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0584283Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0584286Z 2025-11-03T16:41:14.0584383Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0584593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0584653Z res = mod(**inputs) 2025-11-03T16:41:14.0584904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0584981Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0585224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0585300Z outputs = layer_module( 2025-11-03T16:41:14.0585540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0585612Z outputs = self.rel_attn( 2025-11-03T16:41:14.0585849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0585918Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0586184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0586304Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0586307Z 2025-11-03T16:41:14.0586410Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0586598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0586667Z res = mod(**inputs) 2025-11-03T16:41:14.0586927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0587007Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0587255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0587322Z outputs = layer_module( 2025-11-03T16:41:14.0587569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0587635Z outputs = self.rel_attn( 2025-11-03T16:41:14.0587877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0587971Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0588246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0588363Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0588367Z 2025-11-03T16:41:14.0588465Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0588661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0588725Z res = mod(**inputs) 2025-11-03T16:41:14.0588966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0589053Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0589295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0589366Z outputs = layer_module( 2025-11-03T16:41:14.0589625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0589692Z outputs = self.rel_attn( 2025-11-03T16:41:14.0589944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0590028Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0590298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0590419Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0590423Z 2025-11-03T16:41:14.0590528Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0590712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0590773Z res = mod(**inputs) 2025-11-03T16:41:14.0591024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0591102Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0591351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0591416Z outputs = layer_module( 2025-11-03T16:41:14.0591656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0591864Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0592111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0592194Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0592441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0592516Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0592756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0592842Z output = self.layer_1(output) 2025-11-03T16:41:14.0592845Z 2025-11-03T16:41:14.0592951Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0593139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0593211Z res = mod(**inputs) 2025-11-03T16:41:14.0593452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0593530Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0593777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0593867Z outputs = layer_module( 2025-11-03T16:41:14.0594189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0594404Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0594673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0594753Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0595006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0595085Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0595336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0595458Z output = self.activation_function(output) 2025-11-03T16:41:14.0595666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0595738Z return self.act(input) 2025-11-03T16:41:14.0595742Z 2025-11-03T16:41:14.0595849Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0596038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0596108Z res = mod(**inputs) 2025-11-03T16:41:14.0596366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0596447Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0596703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0596769Z outputs = layer_module( 2025-11-03T16:41:14.0597023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0597213Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0597469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0597540Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0597782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0597860Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0598098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0598173Z output = self.layer_2(output) 2025-11-03T16:41:14.0598176Z 2025-11-03T16:41:14.0598272Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0598459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0598527Z res = mod(**inputs) 2025-11-03T16:41:14.0598780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0598863Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0599099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0599170Z outputs = layer_module( 2025-11-03T16:41:14.0599410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0599475Z outputs = self.rel_attn( 2025-11-03T16:41:14.0599719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-11-03T16:41:14.0599830Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-11-03T16:41:14.0599833Z 2025-11-03T16:41:14.0599938Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0600125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0600185Z res = mod(**inputs) 2025-11-03T16:41:14.0600425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0600501Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0600745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0600807Z outputs = layer_module( 2025-11-03T16:41:14.0601042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0601114Z outputs = self.rel_attn( 2025-11-03T16:41:14.0601362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-11-03T16:41:14.0601463Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-11-03T16:41:14.0601468Z 2025-11-03T16:41:14.0601643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0601824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0601890Z res = mod(**inputs) 2025-11-03T16:41:14.0602144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0602221Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0602466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0602529Z outputs = layer_module( 2025-11-03T16:41:14.0602783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0602847Z outputs = self.rel_attn( 2025-11-03T16:41:14.0603084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0603161Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0603411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-11-03T16:41:14.0603543Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-11-03T16:41:14.0603546Z 2025-11-03T16:41:14.0603643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0603836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0603896Z res = mod(**inputs) 2025-11-03T16:41:14.0604142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0604228Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0604467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0604556Z outputs = layer_module( 2025-11-03T16:41:14.0604796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0604862Z outputs = self.rel_attn( 2025-11-03T16:41:14.0605106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-11-03T16:41:14.0605227Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-11-03T16:41:14.0605230Z 2025-11-03T16:41:14.0605331Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0605529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0605590Z res = mod(**inputs) 2025-11-03T16:41:14.0605832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0605909Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0606149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0606212Z outputs = layer_module( 2025-11-03T16:41:14.0606454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0606519Z outputs = self.rel_attn( 2025-11-03T16:41:14.0606752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0606826Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0607092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-11-03T16:41:14.0607219Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-11-03T16:41:14.0607223Z 2025-11-03T16:41:14.0607318Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0607504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0607572Z res = mod(**inputs) 2025-11-03T16:41:14.0607823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0607907Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0608145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0608215Z outputs = layer_module( 2025-11-03T16:41:14.0608449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0608514Z outputs = self.rel_attn( 2025-11-03T16:41:14.0608757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-11-03T16:41:14.0608848Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-11-03T16:41:14.0608852Z 2025-11-03T16:41:14.0608953Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0609134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0609194Z res = mod(**inputs) 2025-11-03T16:41:14.0609436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0609512Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0609756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0609819Z outputs = layer_module( 2025-11-03T16:41:14.0610050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0610137Z outputs = self.rel_attn( 2025-11-03T16:41:14.0610372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-11-03T16:41:14.0610448Z attn_vec = self.rel_attn_core( 2025-11-03T16:41:14.0610703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-11-03T16:41:14.0610825Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-11-03T16:41:14.0610829Z 2025-11-03T16:41:14.0610926Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0611110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0611195Z res = mod(**inputs) 2025-11-03T16:41:14.0611434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0611524Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0611761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0611824Z outputs = layer_module( 2025-11-03T16:41:14.0612066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0612131Z outputs = self.rel_attn( 2025-11-03T16:41:14.0612377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0612461Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0612738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0612843Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0612848Z 2025-11-03T16:41:14.0612943Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0613131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0613189Z res = mod(**inputs) 2025-11-03T16:41:14.0613548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0613670Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0613908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0613980Z outputs = layer_module( 2025-11-03T16:41:14.0614217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-11-03T16:41:14.0614289Z outputs = self.rel_attn( 2025-11-03T16:41:14.0614523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-11-03T16:41:14.0614614Z output_h = self.post_attention(h, attn_vec) 2025-11-03T16:41:14.0614870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-11-03T16:41:14.0614974Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-11-03T16:41:14.0614978Z 2025-11-03T16:41:14.0615082Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0615265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0615333Z res = mod(**inputs) 2025-11-03T16:41:14.0615570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0615647Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0615897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0615991Z outputs = layer_module( 2025-11-03T16:41:14.0616233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0616426Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0616679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0616752Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0616989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0617089Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0617325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-11-03T16:41:14.0617401Z output = self.layer_1(output) 2025-11-03T16:41:14.0617405Z 2025-11-03T16:41:14.0617500Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0617682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0617754Z res = mod(**inputs) 2025-11-03T16:41:14.0617987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0618072Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0618306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0618370Z outputs = layer_module( 2025-11-03T16:41:14.0618632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0618825Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0619080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0619153Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0619408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0619474Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0619710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-11-03T16:41:14.0619796Z output = self.activation_function(output) 2025-11-03T16:41:14.0619997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:14.0620068Z return self.act(input) 2025-11-03T16:41:14.0620072Z 2025-11-03T16:41:14.0620165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0620355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0620414Z res = mod(**inputs) 2025-11-03T16:41:14.0620650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-11-03T16:41:14.0620736Z transformer_outputs = self.transformer( 2025-11-03T16:41:14.0620969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-11-03T16:41:14.0621039Z outputs = layer_module( 2025-11-03T16:41:14.0621274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-11-03T16:41:14.0621465Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-11-03T16:41:14.0621733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:14.0621806Z return forward_fn(*input_tensors) 2025-11-03T16:41:14.0622053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-11-03T16:41:14.0622122Z output_x = self.ff(output_x) 2025-11-03T16:41:14.0622358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-11-03T16:41:14.0622431Z output = self.layer_2(output) 2025-11-03T16:41:14.0622435Z 2025-11-03T16:41:14.0622529Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0622739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0622800Z res = mod(**inputs) 2025-11-03T16:41:14.0623045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1624, in forward 2025-11-03T16:41:14.0623135Z logits = self.lm_loss(transformer_outputs[0]) 2025-11-03T16:41:14.0623138Z 2025-11-03T16:41:14.0623232Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:14.0623423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:14.0623483Z res = mod(**inputs) 2025-11-03T16:41:14.0623726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1630, in forward 2025-11-03T16:41:14.0623849Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-11-03T16:41:14.0623852Z 2025-11-03T16:41:26.9598539Z Compilation time (from dynamo_timed): 32.10569801 2025-11-03T16:41:26.9639036Z pass 2025-11-03T16:41:26.9644705Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:41:26.9649557Z TIMING: _recursive_pre_grad_passes:0.01143 _recursive_joint_graph_passes:1.35126 _recursive_post_grad_passes:0.17222 async_compile.wait:0.99602 code_gen:11.81102 inductor_compile:16.84159 backend_compile:25.96373 gc:0.00033 entire_frame_compile:32.1057 total_wall_time:32.1057 2025-11-03T16:41:26.9651232Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:32619 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:6726 2025-11-03T16:41:26.9651939Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-11-03T16:41:29.9200660Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-11-03T16:41:29.9201474Z import pynvml # type: ignore[import] 2025-11-03T16:41:33.0695614Z 2025-11-03T16:41:34.1264741Z loading model: 0it [00:00, ?it/s] 2025-11-03T16:41:34.1269314Z loading model: 0it [00:01, ?it/s] 2025-11-03T16:41:34.1283828Z cpu eval YituTechConvBert 2025-11-03T16:41:35.0727702Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:41:35.3401494Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:41:35.6214954Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:41:46.9864070Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9870249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9870930Z res = mod(**inputs) 2025-11-03T16:41:46.9871565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9872055Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9872792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9873218Z hidden_states = self.encoder( 2025-11-03T16:41:46.9873642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9874327Z layer_outputs = layer_module( 2025-11-03T16:41:46.9874847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9875314Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9875757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9876308Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9876754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9877162Z self_outputs = self.self( 2025-11-03T16:41:46.9877558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:46.9877984Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:46.9878142Z 2025-11-03T16:41:46.9878264Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9878636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9878961Z res = mod(**inputs) 2025-11-03T16:41:46.9879375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9879923Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9880553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9880965Z hidden_states = self.encoder( 2025-11-03T16:41:46.9881372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9881773Z layer_outputs = layer_module( 2025-11-03T16:41:46.9882144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9882644Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9883049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9883465Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9883886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9884296Z self_outputs = self.self( 2025-11-03T16:41:46.9884689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:46.9885098Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:46.9885247Z 2025-11-03T16:41:46.9885355Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9885724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9886057Z res = mod(**inputs) 2025-11-03T16:41:46.9886439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9886853Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9887270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9887671Z hidden_states = self.encoder( 2025-11-03T16:41:46.9888077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9888592Z layer_outputs = layer_module( 2025-11-03T16:41:46.9888937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9889285Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9889678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9890071Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9890455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9890895Z self_outputs = self.self( 2025-11-03T16:41:46.9891292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:46.9891702Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:46.9891844Z 2025-11-03T16:41:46.9891931Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9892135Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9892379Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9892736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9893093Z res = mod(**inputs) 2025-11-03T16:41:46.9893599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9894176Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9894632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9895017Z hidden_states = self.encoder( 2025-11-03T16:41:46.9895395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9895792Z layer_outputs = layer_module( 2025-11-03T16:41:46.9896282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9896789Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9897374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9897959Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9898527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9899227Z self_outputs = self.self( 2025-11-03T16:41:46.9899607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:46.9900012Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:46.9900236Z 2025-11-03T16:41:46.9900346Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9900643Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9900989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9901301Z res = mod(**inputs) 2025-11-03T16:41:46.9901653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9902044Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9902436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9902822Z hidden_states = self.encoder( 2025-11-03T16:41:46.9903231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9903696Z layer_outputs = layer_module( 2025-11-03T16:41:46.9904093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9904455Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9904842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9905241Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9905655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9906083Z self_outputs = self.self( 2025-11-03T16:41:46.9906546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:46.9907135Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:46.9907619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:46.9908021Z x = self.depthwise(hidden_states) 2025-11-03T16:41:46.9908158Z 2025-11-03T16:41:46.9908260Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9908638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9908977Z res = mod(**inputs) 2025-11-03T16:41:46.9909344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9909748Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9910188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9910716Z hidden_states = self.encoder( 2025-11-03T16:41:46.9911107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9911508Z layer_outputs = layer_module( 2025-11-03T16:41:46.9911849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9912223Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9912620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9913019Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9913722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9914259Z self_outputs = self.self( 2025-11-03T16:41:46.9914815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:46.9915354Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:46.9915850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:46.9916247Z x = self.pointwise(x) 2025-11-03T16:41:46.9916366Z 2025-11-03T16:41:46.9916468Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9916828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9917148Z res = mod(**inputs) 2025-11-03T16:41:46.9917511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9917917Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9918487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9919043Z hidden_states = self.encoder( 2025-11-03T16:41:46.9919428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9919843Z layer_outputs = layer_module( 2025-11-03T16:41:46.9920186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9920546Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9920939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9921366Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9921793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9922195Z self_outputs = self.self( 2025-11-03T16:41:46.9922663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:46.9923236Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:46.9923438Z 2025-11-03T16:41:46.9923547Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9923894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9924210Z res = mod(**inputs) 2025-11-03T16:41:46.9924588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9924992Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9925436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9925822Z hidden_states = self.encoder( 2025-11-03T16:41:46.9926197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9926576Z layer_outputs = layer_module( 2025-11-03T16:41:46.9926997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9927542Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9928079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9928647Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9929103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9929501Z self_outputs = self.self( 2025-11-03T16:41:46.9929993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:46.9930546Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:46.9930791Z 2025-11-03T16:41:46.9930895Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9931333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9931768Z res = mod(**inputs) 2025-11-03T16:41:46.9932245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9932640Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9933042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9933427Z hidden_states = self.encoder( 2025-11-03T16:41:46.9933799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9934210Z layer_outputs = layer_module( 2025-11-03T16:41:46.9934545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9934896Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9935313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9935842Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9936231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9936613Z self_outputs = self.self( 2025-11-03T16:41:46.9937005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:46.9937442Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:46.9937613Z 2025-11-03T16:41:46.9937692Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9937901Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9938130Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9938473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9938776Z res = mod(**inputs) 2025-11-03T16:41:46.9939138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9939535Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9939929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9940344Z hidden_states = self.encoder( 2025-11-03T16:41:46.9940716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9941101Z layer_outputs = layer_module( 2025-11-03T16:41:46.9941436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9941785Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9942166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9942575Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9942965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9943344Z self_outputs = self.self( 2025-11-03T16:41:46.9943720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:46.9944136Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:46.9944303Z 2025-11-03T16:41:46.9944404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9944745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9945050Z res = mod(**inputs) 2025-11-03T16:41:46.9945409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9945796Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9946189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9946569Z hidden_states = self.encoder( 2025-11-03T16:41:46.9946946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9947328Z layer_outputs = layer_module( 2025-11-03T16:41:46.9947652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9948017Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9948406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9948799Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9949188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:46.9949628Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:46.9950063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:46.9950462Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:46.9950612Z 2025-11-03T16:41:46.9950722Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9951061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9951370Z res = mod(**inputs) 2025-11-03T16:41:46.9951732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9952132Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9952530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9952908Z hidden_states = self.encoder( 2025-11-03T16:41:46.9953287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9953670Z layer_outputs = layer_module( 2025-11-03T16:41:46.9954038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9955240Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9955682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:46.9956081Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:46.9956474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:46.9956912Z return forward_fn(*input_tensors) 2025-11-03T16:41:46.9957366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:46.9957887Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:46.9958378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:46.9958818Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:46.9958970Z 2025-11-03T16:41:46.9959092Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9959467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9959810Z res = mod(**inputs) 2025-11-03T16:41:46.9960214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9960652Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9961099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9961529Z hidden_states = self.encoder( 2025-11-03T16:41:46.9961966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9962388Z layer_outputs = layer_module( 2025-11-03T16:41:46.9962760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9963159Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9963573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:46.9963964Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:46.9964349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:46.9964725Z return forward_fn(*input_tensors) 2025-11-03T16:41:46.9965129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:46.9965605Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:46.9966034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:46.9966458Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:46.9966824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:46.9967152Z return self.act(input) 2025-11-03T16:41:46.9967268Z 2025-11-03T16:41:46.9967371Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9967718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9968026Z res = mod(**inputs) 2025-11-03T16:41:46.9968391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9968781Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9969195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9969579Z hidden_states = self.encoder( 2025-11-03T16:41:46.9969957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9970326Z layer_outputs = layer_module( 2025-11-03T16:41:46.9970661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9971849Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9972247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:46.9972645Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:46.9973034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:46.9973423Z return forward_fn(*input_tensors) 2025-11-03T16:41:46.9973843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:46.9974319Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:46.9974766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:46.9975158Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:46.9975301Z 2025-11-03T16:41:46.9975405Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9975753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9976065Z res = mod(**inputs) 2025-11-03T16:41:46.9976429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9976833Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9977231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9977633Z hidden_states = self.encoder( 2025-11-03T16:41:46.9978005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9978379Z layer_outputs = layer_module( 2025-11-03T16:41:46.9978713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9979057Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9979442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9979840Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9980234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9980618Z self_outputs = self.self( 2025-11-03T16:41:46.9980995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:46.9981402Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:46.9981544Z 2025-11-03T16:41:46.9981651Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9981989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9982305Z res = mod(**inputs) 2025-11-03T16:41:46.9982683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9983098Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9983529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9983925Z hidden_states = self.encoder( 2025-11-03T16:41:46.9984311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9984713Z layer_outputs = layer_module( 2025-11-03T16:41:46.9985061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9985439Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9985847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9986260Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9986679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9987079Z self_outputs = self.self( 2025-11-03T16:41:46.9987462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:46.9987875Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:46.9988019Z 2025-11-03T16:41:46.9988123Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9988481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9988798Z res = mod(**inputs) 2025-11-03T16:41:46.9989174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9989584Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9989992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9990391Z hidden_states = self.encoder( 2025-11-03T16:41:46.9990795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9991218Z layer_outputs = layer_module( 2025-11-03T16:41:46.9991569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9991926Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9992303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9992677Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9993058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:46.9993428Z self_outputs = self.self( 2025-11-03T16:41:46.9993821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:46.9994305Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:46.9994468Z 2025-11-03T16:41:46.9994553Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9994773Z cudagraph partition due to non gpu ops 2025-11-03T16:41:46.9995013Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:46.9995375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:46.9995701Z res = mod(**inputs) 2025-11-03T16:41:46.9996061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:46.9996448Z generator_hidden_states = self.convbert( 2025-11-03T16:41:46.9996837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:46.9997231Z hidden_states = self.encoder( 2025-11-03T16:41:46.9997593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:46.9997965Z layer_outputs = layer_module( 2025-11-03T16:41:46.9998284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:46.9998618Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:46.9998985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:46.9999385Z self_attention_outputs = self.attention( 2025-11-03T16:41:46.9999770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0000148Z self_outputs = self.self( 2025-11-03T16:41:47.0000516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0000914Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0001068Z 2025-11-03T16:41:47.0001142Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0001361Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0001697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0001993Z res = mod(**inputs) 2025-11-03T16:41:47.0002346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0002734Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0003119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0003493Z hidden_states = self.encoder( 2025-11-03T16:41:47.0003859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0004235Z layer_outputs = layer_module( 2025-11-03T16:41:47.0004575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0004915Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0005298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0005684Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0006069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0006446Z self_outputs = self.self( 2025-11-03T16:41:47.0006816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0007297Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0007756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0008140Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0008270Z 2025-11-03T16:41:47.0008368Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0008708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0009001Z res = mod(**inputs) 2025-11-03T16:41:47.0009351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0009733Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0010133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0010509Z hidden_states = self.encoder( 2025-11-03T16:41:47.0010873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0011245Z layer_outputs = layer_module( 2025-11-03T16:41:47.0011567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0011905Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0012302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0012679Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0013070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0013584Z self_outputs = self.self( 2025-11-03T16:41:47.0013963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0014420Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0014891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0015273Z x = self.pointwise(x) 2025-11-03T16:41:47.0015387Z 2025-11-03T16:41:47.0015487Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0015835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0016139Z res = mod(**inputs) 2025-11-03T16:41:47.0016503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0016885Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0017269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0017643Z hidden_states = self.encoder( 2025-11-03T16:41:47.0018062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0018439Z layer_outputs = layer_module( 2025-11-03T16:41:47.0018771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0019121Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0019508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0019893Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0020284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0020690Z self_outputs = self.self( 2025-11-03T16:41:47.0021055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0021498Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0021701Z 2025-11-03T16:41:47.0021798Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0022133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0022435Z res = mod(**inputs) 2025-11-03T16:41:47.0022782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0023156Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0023560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0023941Z hidden_states = self.encoder( 2025-11-03T16:41:47.0024314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0024686Z layer_outputs = layer_module( 2025-11-03T16:41:47.0025006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0025347Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0025755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0026148Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0026536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0026918Z self_outputs = self.self( 2025-11-03T16:41:47.0027294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0027723Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0027888Z 2025-11-03T16:41:47.0027996Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0028332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0028644Z res = mod(**inputs) 2025-11-03T16:41:47.0029005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0029397Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0029786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0030159Z hidden_states = self.encoder( 2025-11-03T16:41:47.0030537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0030913Z layer_outputs = layer_module( 2025-11-03T16:41:47.0031250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0031586Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0031956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0032336Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0032714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0033080Z self_outputs = self.self( 2025-11-03T16:41:47.0033433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0033880Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0034054Z 2025-11-03T16:41:47.0034129Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0034497Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0034833Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0035435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0035761Z res = mod(**inputs) 2025-11-03T16:41:47.0036124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0036511Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0036908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0037306Z hidden_states = self.encoder( 2025-11-03T16:41:47.0037737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0038132Z layer_outputs = layer_module( 2025-11-03T16:41:47.0038478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0038830Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0039231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0039659Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0040061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0040449Z self_outputs = self.self( 2025-11-03T16:41:47.0040830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0041268Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0041441Z 2025-11-03T16:41:47.0041544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0041899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0042215Z res = mod(**inputs) 2025-11-03T16:41:47.0042577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0042981Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0043381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0043782Z hidden_states = self.encoder( 2025-11-03T16:41:47.0044162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0044558Z layer_outputs = layer_module( 2025-11-03T16:41:47.0044901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0045296Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0045694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0046065Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0046447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0046873Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0047294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0047681Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0047812Z 2025-11-03T16:41:47.0047930Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0048269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0048572Z res = mod(**inputs) 2025-11-03T16:41:47.0048919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0049293Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0049671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0050039Z hidden_states = self.encoder( 2025-11-03T16:41:47.0050409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0050777Z layer_outputs = layer_module( 2025-11-03T16:41:47.0051109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0051447Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0051822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0052208Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0052585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0052961Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0053364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0053809Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0054226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0054612Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0054741Z 2025-11-03T16:41:47.0054838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0055175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0055476Z res = mod(**inputs) 2025-11-03T16:41:47.0055825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0056202Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0056585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0056957Z hidden_states = self.encoder( 2025-11-03T16:41:47.0057323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0057691Z layer_outputs = layer_module( 2025-11-03T16:41:47.0058023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0058376Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0058749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0059133Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0059510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0059868Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0060267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0060713Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0061146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0061556Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0061915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0062252Z return self.act(input) 2025-11-03T16:41:47.0062361Z 2025-11-03T16:41:47.0062470Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0062816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0063117Z res = mod(**inputs) 2025-11-03T16:41:47.0063475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0063868Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0064277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0064661Z hidden_states = self.encoder( 2025-11-03T16:41:47.0065030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0065409Z layer_outputs = layer_module( 2025-11-03T16:41:47.0065740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0066106Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0066492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0066875Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0067245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0067617Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0068010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0068456Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0068881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0069260Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0069387Z 2025-11-03T16:41:47.0069489Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0069824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0070116Z res = mod(**inputs) 2025-11-03T16:41:47.0070466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0070848Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0071230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0071622Z hidden_states = self.encoder( 2025-11-03T16:41:47.0071984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0072356Z layer_outputs = layer_module( 2025-11-03T16:41:47.0072685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0073028Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0073397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0073781Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0074275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0074680Z self_outputs = self.self( 2025-11-03T16:41:47.0075068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0075472Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0075624Z 2025-11-03T16:41:47.0075727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0076082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0076397Z res = mod(**inputs) 2025-11-03T16:41:47.0076768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0077164Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0077591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0077985Z hidden_states = self.encoder( 2025-11-03T16:41:47.0078370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0078760Z layer_outputs = layer_module( 2025-11-03T16:41:47.0079095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0079449Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0079870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0080275Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0080671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0081069Z self_outputs = self.self( 2025-11-03T16:41:47.0081454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0081864Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0081999Z 2025-11-03T16:41:47.0082106Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0082456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0082775Z res = mod(**inputs) 2025-11-03T16:41:47.0083145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0083555Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0083959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0084347Z hidden_states = self.encoder( 2025-11-03T16:41:47.0084741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0085147Z layer_outputs = layer_module( 2025-11-03T16:41:47.0085488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0085839Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0086237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0086646Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0087043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0087431Z self_outputs = self.self( 2025-11-03T16:41:47.0087808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0088236Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0088390Z 2025-11-03T16:41:47.0088470Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0088687Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0088914Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0089269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0089585Z res = mod(**inputs) 2025-11-03T16:41:47.0089960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0090366Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0090762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0091154Z hidden_states = self.encoder( 2025-11-03T16:41:47.0091557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0091952Z layer_outputs = layer_module( 2025-11-03T16:41:47.0092294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0092642Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0093040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0093470Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0093869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0094252Z self_outputs = self.self( 2025-11-03T16:41:47.0094626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0095026Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0095171Z 2025-11-03T16:41:47.0095252Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0095475Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0095802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0096102Z res = mod(**inputs) 2025-11-03T16:41:47.0096449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0096833Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0097214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0097575Z hidden_states = self.encoder( 2025-11-03T16:41:47.0097942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0098309Z layer_outputs = layer_module( 2025-11-03T16:41:47.0098629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0098987Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0099369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0099753Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0100141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0100516Z self_outputs = self.self( 2025-11-03T16:41:47.0100877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0101357Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0101816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0102198Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0102318Z 2025-11-03T16:41:47.0102423Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0102752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0103056Z res = mod(**inputs) 2025-11-03T16:41:47.0103406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0103788Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0104173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0104571Z hidden_states = self.encoder( 2025-11-03T16:41:47.0104952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0105338Z layer_outputs = layer_module( 2025-11-03T16:41:47.0105681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0106025Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0106417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0106832Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0107227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0107612Z self_outputs = self.self( 2025-11-03T16:41:47.0107983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0108456Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0108925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0109310Z x = self.pointwise(x) 2025-11-03T16:41:47.0109414Z 2025-11-03T16:41:47.0109519Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0109857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0110166Z res = mod(**inputs) 2025-11-03T16:41:47.0110526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0110924Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0111308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0111693Z hidden_states = self.encoder( 2025-11-03T16:41:47.0112071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0112470Z layer_outputs = layer_module( 2025-11-03T16:41:47.0112804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0113146Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0113813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0114328Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0114955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0115440Z self_outputs = self.self( 2025-11-03T16:41:47.0115866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0116347Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0116553Z 2025-11-03T16:41:47.0116654Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0117001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0117312Z res = mod(**inputs) 2025-11-03T16:41:47.0117662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0118055Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0118446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0118853Z hidden_states = self.encoder( 2025-11-03T16:41:47.0119225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0119606Z layer_outputs = layer_module( 2025-11-03T16:41:47.0119938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0120286Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0120672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0121088Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0121489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0121871Z self_outputs = self.self( 2025-11-03T16:41:47.0122249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0122681Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0122848Z 2025-11-03T16:41:47.0122946Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0123293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0123605Z res = mod(**inputs) 2025-11-03T16:41:47.0123970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0124358Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0124761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0125148Z hidden_states = self.encoder( 2025-11-03T16:41:47.0125534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0125917Z layer_outputs = layer_module( 2025-11-03T16:41:47.0126248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0126625Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0127015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0127404Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0127796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0128168Z self_outputs = self.self( 2025-11-03T16:41:47.0128538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0128972Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0129162Z 2025-11-03T16:41:47.0129251Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0129459Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0129688Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0130041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0130352Z res = mod(**inputs) 2025-11-03T16:41:47.0130714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0131105Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0131500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0131881Z hidden_states = self.encoder( 2025-11-03T16:41:47.0132276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0132654Z layer_outputs = layer_module( 2025-11-03T16:41:47.0132979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0133324Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0133711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0134099Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0134493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0134869Z self_outputs = self.self( 2025-11-03T16:41:47.0135237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0135655Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0135811Z 2025-11-03T16:41:47.0135916Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0136249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0136557Z res = mod(**inputs) 2025-11-03T16:41:47.0136910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0137294Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0137683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0138058Z hidden_states = self.encoder( 2025-11-03T16:41:47.0138431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0138810Z layer_outputs = layer_module( 2025-11-03T16:41:47.0139139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0139468Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0139865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0140247Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0140629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0141052Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0141464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0141849Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0141984Z 2025-11-03T16:41:47.0142082Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0142425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0142726Z res = mod(**inputs) 2025-11-03T16:41:47.0143065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0143446Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0143825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0144196Z hidden_states = self.encoder( 2025-11-03T16:41:47.0144560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0144925Z layer_outputs = layer_module( 2025-11-03T16:41:47.0145320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0145661Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0146039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0146422Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0146799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0147166Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0147595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0148040Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0148451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0148838Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0148974Z 2025-11-03T16:41:47.0149070Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0149407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0149704Z res = mod(**inputs) 2025-11-03T16:41:47.0150049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0150430Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0150811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0151178Z hidden_states = self.encoder( 2025-11-03T16:41:47.0151548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0151910Z layer_outputs = layer_module( 2025-11-03T16:41:47.0152235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0152568Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0152963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0153339Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0153711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0154102Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0154619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0155109Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0155573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0155988Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0156354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0156682Z return self.act(input) 2025-11-03T16:41:47.0156788Z 2025-11-03T16:41:47.0156897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0157227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0157533Z res = mod(**inputs) 2025-11-03T16:41:47.0157886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0158267Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0158655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0159034Z hidden_states = self.encoder( 2025-11-03T16:41:47.0159410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0159804Z layer_outputs = layer_module( 2025-11-03T16:41:47.0160129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0160461Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0160864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0161269Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0161668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0162071Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0162476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0162941Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0163417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0163844Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0163976Z 2025-11-03T16:41:47.0164088Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0164427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0164735Z res = mod(**inputs) 2025-11-03T16:41:47.0165099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0165495Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0165886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0166286Z hidden_states = self.encoder( 2025-11-03T16:41:47.0166665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0167046Z layer_outputs = layer_module( 2025-11-03T16:41:47.0167376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0167718Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0168103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0168503Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0168906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0169289Z self_outputs = self.self( 2025-11-03T16:41:47.0169658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0170064Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0170211Z 2025-11-03T16:41:47.0170313Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0170653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0170956Z res = mod(**inputs) 2025-11-03T16:41:47.0171314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0171704Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0172111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0172496Z hidden_states = self.encoder( 2025-11-03T16:41:47.0172870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0173255Z layer_outputs = layer_module( 2025-11-03T16:41:47.0173589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0173939Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0174346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0174734Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0175125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0175509Z self_outputs = self.self( 2025-11-03T16:41:47.0175921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0176320Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0176466Z 2025-11-03T16:41:47.0176566Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0176917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0177231Z res = mod(**inputs) 2025-11-03T16:41:47.0177597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0177993Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0178394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0178787Z hidden_states = self.encoder( 2025-11-03T16:41:47.0179175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0179568Z layer_outputs = layer_module( 2025-11-03T16:41:47.0179925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0180283Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0180683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0181091Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0181485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0181882Z self_outputs = self.self( 2025-11-03T16:41:47.0182267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0182703Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0182848Z 2025-11-03T16:41:47.0182938Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0183152Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0183396Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0183770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0184080Z res = mod(**inputs) 2025-11-03T16:41:47.0184444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0184839Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0185247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0185643Z hidden_states = self.encoder( 2025-11-03T16:41:47.0186048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0186436Z layer_outputs = layer_module( 2025-11-03T16:41:47.0186779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0187134Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0187536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0187951Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0188345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0188734Z self_outputs = self.self( 2025-11-03T16:41:47.0189113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0189535Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0189688Z 2025-11-03T16:41:47.0189774Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0190003Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0190354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0190670Z res = mod(**inputs) 2025-11-03T16:41:47.0191037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0191432Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0191835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0192231Z hidden_states = self.encoder( 2025-11-03T16:41:47.0192596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0192967Z layer_outputs = layer_module( 2025-11-03T16:41:47.0193292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0193651Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0194035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0194544Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0194956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0195337Z self_outputs = self.self( 2025-11-03T16:41:47.0195711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0196168Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0196656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0197049Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0197172Z 2025-11-03T16:41:47.0197269Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0197606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0197908Z res = mod(**inputs) 2025-11-03T16:41:47.0198262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0198643Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0199031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0199405Z hidden_states = self.encoder( 2025-11-03T16:41:47.0199794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0200165Z layer_outputs = layer_module( 2025-11-03T16:41:47.0200482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0200818Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0201189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0201587Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0201959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0202328Z self_outputs = self.self( 2025-11-03T16:41:47.0202684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0203135Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0203591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0203956Z x = self.pointwise(x) 2025-11-03T16:41:47.0204067Z 2025-11-03T16:41:47.0204163Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0204495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0204797Z res = mod(**inputs) 2025-11-03T16:41:47.0205146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0205522Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0205905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0206277Z hidden_states = self.encoder( 2025-11-03T16:41:47.0206645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0207037Z layer_outputs = layer_module( 2025-11-03T16:41:47.0207355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0207699Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0208082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0208465Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0208842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0209215Z self_outputs = self.self( 2025-11-03T16:41:47.0209594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0210049Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0210244Z 2025-11-03T16:41:47.0210352Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0210683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0210985Z res = mod(**inputs) 2025-11-03T16:41:47.0211342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0211731Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0212116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0212484Z hidden_states = self.encoder( 2025-11-03T16:41:47.0212875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0213355Z layer_outputs = layer_module( 2025-11-03T16:41:47.0213696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0214038Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0214413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0214842Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0215218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0215585Z self_outputs = self.self( 2025-11-03T16:41:47.0215937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0216353Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0216518Z 2025-11-03T16:41:47.0216615Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0216949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0217248Z res = mod(**inputs) 2025-11-03T16:41:47.0217591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0217976Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0218354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0218724Z hidden_states = self.encoder( 2025-11-03T16:41:47.0219088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0219454Z layer_outputs = layer_module( 2025-11-03T16:41:47.0219779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0220145Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0220525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0220902Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0221283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0221656Z self_outputs = self.self( 2025-11-03T16:41:47.0222018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0222444Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0222610Z 2025-11-03T16:41:47.0222713Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0222917Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0223141Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0223476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0223775Z res = mod(**inputs) 2025-11-03T16:41:47.0224121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0224507Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0224890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0225259Z hidden_states = self.encoder( 2025-11-03T16:41:47.0225617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0226018Z layer_outputs = layer_module( 2025-11-03T16:41:47.0226345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0226713Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0227094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0227466Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0227868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0228242Z self_outputs = self.self( 2025-11-03T16:41:47.0228607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0229020Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0229174Z 2025-11-03T16:41:47.0229271Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0229607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0229911Z res = mod(**inputs) 2025-11-03T16:41:47.0230260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0230636Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0231020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0231391Z hidden_states = self.encoder( 2025-11-03T16:41:47.0231760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0232130Z layer_outputs = layer_module( 2025-11-03T16:41:47.0232446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0232786Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0233159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0233554Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0233930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0234536Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0235152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0235559Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0235689Z 2025-11-03T16:41:47.0235792Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0236159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0236458Z res = mod(**inputs) 2025-11-03T16:41:47.0236811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0237196Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0237593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0237973Z hidden_states = self.encoder( 2025-11-03T16:41:47.0238354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0238736Z layer_outputs = layer_module( 2025-11-03T16:41:47.0239071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0239443Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0239827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0240226Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0240615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0240992Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0241405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0241873Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0242303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0242703Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0242835Z 2025-11-03T16:41:47.0242944Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0243281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0243592Z res = mod(**inputs) 2025-11-03T16:41:47.0243950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0244345Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0244738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0245115Z hidden_states = self.encoder( 2025-11-03T16:41:47.0245493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0245880Z layer_outputs = layer_module( 2025-11-03T16:41:47.0246216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0246565Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0246982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0247383Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0247771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0248155Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0248562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0249022Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0249456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0249888Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0250250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0250566Z return self.act(input) 2025-11-03T16:41:47.0250679Z 2025-11-03T16:41:47.0250776Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0251112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0251414Z res = mod(**inputs) 2025-11-03T16:41:47.0251762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0252138Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0252520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0252913Z hidden_states = self.encoder( 2025-11-03T16:41:47.0253280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0253651Z layer_outputs = layer_module( 2025-11-03T16:41:47.0253971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0254307Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0254681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0255082Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0255448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0255812Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0256214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0256673Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0257100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0257473Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0257610Z 2025-11-03T16:41:47.0257711Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0258048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0258352Z res = mod(**inputs) 2025-11-03T16:41:47.0258697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0259071Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0259329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0259397Z hidden_states = self.encoder( 2025-11-03T16:41:47.0259669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0259736Z layer_outputs = layer_module( 2025-11-03T16:41:47.0259946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0260029Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0260275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0260367Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0260612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0260704Z self_outputs = self.self( 2025-11-03T16:41:47.0260953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0261042Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0261046Z 2025-11-03T16:41:47.0261149Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0261335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0261404Z res = mod(**inputs) 2025-11-03T16:41:47.0261651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0261725Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0261981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0262062Z hidden_states = self.encoder( 2025-11-03T16:41:47.0262319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0262386Z layer_outputs = layer_module( 2025-11-03T16:41:47.0262590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0262670Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0262916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0263018Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0263266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0263343Z self_outputs = self.self( 2025-11-03T16:41:47.0263594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0263673Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0263676Z 2025-11-03T16:41:47.0263783Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0263968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0264038Z res = mod(**inputs) 2025-11-03T16:41:47.0264285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0264362Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0264614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0264680Z hidden_states = self.encoder( 2025-11-03T16:41:47.0264939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0265006Z layer_outputs = layer_module( 2025-11-03T16:41:47.0265220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0265317Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0265567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0265649Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0265898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0265967Z self_outputs = self.self( 2025-11-03T16:41:47.0266213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0266298Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0266309Z 2025-11-03T16:41:47.0266398Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0266472Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0266601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0266784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0266843Z res = mod(**inputs) 2025-11-03T16:41:47.0267097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0267175Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0267433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0267499Z hidden_states = self.encoder( 2025-11-03T16:41:47.0267770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0267836Z layer_outputs = layer_module( 2025-11-03T16:41:47.0268042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0268126Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0268381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0268463Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0268724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0268789Z self_outputs = self.self( 2025-11-03T16:41:47.0269045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0269140Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0269143Z 2025-11-03T16:41:47.0269227Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0269322Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0269510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0269570Z res = mod(**inputs) 2025-11-03T16:41:47.0269823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0269905Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0270154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0270226Z hidden_states = self.encoder( 2025-11-03T16:41:47.0270472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0270538Z layer_outputs = layer_module( 2025-11-03T16:41:47.0270753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0270844Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0271100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0271175Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0271436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0271502Z self_outputs = self.self( 2025-11-03T16:41:47.0271754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0271914Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0272181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0272262Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0272266Z 2025-11-03T16:41:47.0272362Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0272543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0272610Z res = mod(**inputs) 2025-11-03T16:41:47.0272859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0272940Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0273186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0273258Z hidden_states = self.encoder( 2025-11-03T16:41:47.0273522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0273590Z layer_outputs = layer_module( 2025-11-03T16:41:47.0273802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0273875Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0274138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0274312Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0274615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0274695Z self_outputs = self.self( 2025-11-03T16:41:47.0274956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0275122Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0275382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0275462Z x = self.pointwise(x) 2025-11-03T16:41:47.0275466Z 2025-11-03T16:41:47.0275568Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0275760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0275833Z res = mod(**inputs) 2025-11-03T16:41:47.0276095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0276179Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0276437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0276504Z hidden_states = self.encoder( 2025-11-03T16:41:47.0276774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0276861Z layer_outputs = layer_module( 2025-11-03T16:41:47.0277088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0277162Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0277426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0277515Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0277777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0277852Z self_outputs = self.self( 2025-11-03T16:41:47.0278135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0278293Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0278298Z 2025-11-03T16:41:47.0278400Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0278594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0278665Z res = mod(**inputs) 2025-11-03T16:41:47.0278930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0279018Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0279278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0279345Z hidden_states = self.encoder( 2025-11-03T16:41:47.0279632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0279700Z layer_outputs = layer_module( 2025-11-03T16:41:47.0279923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0279999Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0280265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0280344Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0280633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0280706Z self_outputs = self.self( 2025-11-03T16:41:47.0280971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0281098Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0281102Z 2025-11-03T16:41:47.0281202Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0281397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0281466Z res = mod(**inputs) 2025-11-03T16:41:47.0281728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0281814Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0282080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0282155Z hidden_states = self.encoder( 2025-11-03T16:41:47.0282416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0282484Z layer_outputs = layer_module( 2025-11-03T16:41:47.0282711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0282789Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0283073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0283152Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0283412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0283487Z self_outputs = self.self( 2025-11-03T16:41:47.0283749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0283877Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0283881Z 2025-11-03T16:41:47.0283958Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0284058Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0284160Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0284354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0284423Z res = mod(**inputs) 2025-11-03T16:41:47.0284685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0284770Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0285036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0285105Z hidden_states = self.encoder( 2025-11-03T16:41:47.0285376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0285443Z layer_outputs = layer_module( 2025-11-03T16:41:47.0285684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0285763Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0286026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0286113Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0286376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0286471Z self_outputs = self.self( 2025-11-03T16:41:47.0286734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0286851Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0286855Z 2025-11-03T16:41:47.0286958Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0287163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0287236Z res = mod(**inputs) 2025-11-03T16:41:47.0287498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0287585Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0287848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0287918Z hidden_states = self.encoder( 2025-11-03T16:41:47.0288188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0288257Z layer_outputs = layer_module( 2025-11-03T16:41:47.0288481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0288559Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0288829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0288925Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0289191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0289327Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0289588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0289677Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0289681Z 2025-11-03T16:41:47.0289781Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0289978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0290063Z res = mod(**inputs) 2025-11-03T16:41:47.0290325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0290411Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0290663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0290738Z hidden_states = self.encoder( 2025-11-03T16:41:47.0290987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0291056Z layer_outputs = layer_module( 2025-11-03T16:41:47.0291271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0291346Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0291616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0291697Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0291942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0292023Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0292303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0292444Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0292690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0292773Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0292777Z 2025-11-03T16:41:47.0292871Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0293056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0293124Z res = mod(**inputs) 2025-11-03T16:41:47.0293370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0293452Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0293700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0293769Z hidden_states = self.encoder( 2025-11-03T16:41:47.0294024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0294089Z layer_outputs = layer_module( 2025-11-03T16:41:47.0294301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0294375Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0294630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0294724Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0294965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0295044Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0295327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0295445Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0295695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0295833Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0296040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0296106Z return self.act(input) 2025-11-03T16:41:47.0296110Z 2025-11-03T16:41:47.0296211Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0296393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0296458Z res = mod(**inputs) 2025-11-03T16:41:47.0296706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0296780Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0297034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0297099Z hidden_states = self.encoder( 2025-11-03T16:41:47.0297384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0297450Z layer_outputs = layer_module( 2025-11-03T16:41:47.0297659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0297741Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0297988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0298089Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0298331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0298402Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0298686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0298810Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0299061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0299139Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0299142Z 2025-11-03T16:41:47.0299246Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0299427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0299488Z res = mod(**inputs) 2025-11-03T16:41:47.0299741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0299816Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0300070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0300135Z hidden_states = self.encoder( 2025-11-03T16:41:47.0300390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0300470Z layer_outputs = layer_module( 2025-11-03T16:41:47.0300673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0300753Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0301000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0301085Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0301330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0301396Z self_outputs = self.self( 2025-11-03T16:41:47.0301698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0301789Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0301793Z 2025-11-03T16:41:47.0301897Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0302081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0302140Z res = mod(**inputs) 2025-11-03T16:41:47.0302395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0302471Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0302727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0302796Z hidden_states = self.encoder( 2025-11-03T16:41:47.0303067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0303132Z layer_outputs = layer_module( 2025-11-03T16:41:47.0303341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0303420Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0303666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0303765Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0304016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0304080Z self_outputs = self.self( 2025-11-03T16:41:47.0304343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0304422Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0304426Z 2025-11-03T16:41:47.0304530Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0304716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0304787Z res = mod(**inputs) 2025-11-03T16:41:47.0305040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0305116Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0305376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0305441Z hidden_states = self.encoder( 2025-11-03T16:41:47.0305697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0305761Z layer_outputs = layer_module( 2025-11-03T16:41:47.0305971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0306065Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0306322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0306404Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0306653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0306727Z self_outputs = self.self( 2025-11-03T16:41:47.0306980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0307065Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0307069Z 2025-11-03T16:41:47.0307153Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0307242Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0307345Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0307532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0307591Z res = mod(**inputs) 2025-11-03T16:41:47.0307849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0307924Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0308178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0308243Z hidden_states = self.encoder( 2025-11-03T16:41:47.0308488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0308576Z layer_outputs = layer_module( 2025-11-03T16:41:47.0308784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0308866Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0309115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0309196Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0309442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0309523Z self_outputs = self.self( 2025-11-03T16:41:47.0309775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0309871Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0309875Z 2025-11-03T16:41:47.0309955Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0310051Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0310231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0310300Z res = mod(**inputs) 2025-11-03T16:41:47.0310579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0310662Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0310918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0310992Z hidden_states = self.encoder( 2025-11-03T16:41:47.0311246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0311311Z layer_outputs = layer_module( 2025-11-03T16:41:47.0311529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0311602Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0311882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0311958Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0312219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0312295Z self_outputs = self.self( 2025-11-03T16:41:47.0312549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0312712Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0312986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0313070Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0313074Z 2025-11-03T16:41:47.0313172Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0313576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0313664Z res = mod(**inputs) 2025-11-03T16:41:47.0313932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0314026Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0314368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0314446Z hidden_states = self.encoder( 2025-11-03T16:41:47.0314724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0314849Z layer_outputs = layer_module( 2025-11-03T16:41:47.0315092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0315174Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0315433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0315522Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0315799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0315874Z self_outputs = self.self( 2025-11-03T16:41:47.0316133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0316294Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0316551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0316621Z x = self.pointwise(x) 2025-11-03T16:41:47.0316624Z 2025-11-03T16:41:47.0316733Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0316920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0316991Z res = mod(**inputs) 2025-11-03T16:41:47.0317248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0317327Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0317589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0317657Z hidden_states = self.encoder( 2025-11-03T16:41:47.0317919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0317996Z layer_outputs = layer_module( 2025-11-03T16:41:47.0318237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0318309Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0318562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0318648Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0318902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0318976Z self_outputs = self.self( 2025-11-03T16:41:47.0319230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0319411Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0319421Z 2025-11-03T16:41:47.0319520Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0319711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0319779Z res = mod(**inputs) 2025-11-03T16:41:47.0320036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0320121Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0320386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0320452Z hidden_states = self.encoder( 2025-11-03T16:41:47.0320717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0320798Z layer_outputs = layer_module( 2025-11-03T16:41:47.0321022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0321096Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0321349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0321434Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0321708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0321782Z self_outputs = self.self( 2025-11-03T16:41:47.0322038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0322160Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0322163Z 2025-11-03T16:41:47.0322265Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0322453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0322524Z res = mod(**inputs) 2025-11-03T16:41:47.0322778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0322861Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0323119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0323185Z hidden_states = self.encoder( 2025-11-03T16:41:47.0323452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0323517Z layer_outputs = layer_module( 2025-11-03T16:41:47.0323737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0323812Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0324090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0324165Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0324420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0324495Z self_outputs = self.self( 2025-11-03T16:41:47.0324759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0324888Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0324891Z 2025-11-03T16:41:47.0324966Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0325318Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0325437Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0325620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0325688Z res = mod(**inputs) 2025-11-03T16:41:47.0325938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0326022Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0326268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0326336Z hidden_states = self.encoder( 2025-11-03T16:41:47.0326590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0326654Z layer_outputs = layer_module( 2025-11-03T16:41:47.0326882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0326955Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0327203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0327285Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0327530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0327618Z self_outputs = self.self( 2025-11-03T16:41:47.0327866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0327969Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0327979Z 2025-11-03T16:41:47.0328075Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0328258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0328326Z res = mod(**inputs) 2025-11-03T16:41:47.0328573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0328655Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0328903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0328967Z hidden_states = self.encoder( 2025-11-03T16:41:47.0329220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0329284Z layer_outputs = layer_module( 2025-11-03T16:41:47.0329491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0329565Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0329813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0329915Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0330168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0330298Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0330546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0330633Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0330636Z 2025-11-03T16:41:47.0330733Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0330916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0330984Z res = mod(**inputs) 2025-11-03T16:41:47.0331246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0331331Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0331576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0331643Z hidden_states = self.encoder( 2025-11-03T16:41:47.0331898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0331964Z layer_outputs = layer_module( 2025-11-03T16:41:47.0332172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0332243Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0332512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0332592Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0332837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0332914Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0333195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0333329Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0333577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0333654Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0333665Z 2025-11-03T16:41:47.0333761Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0333947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0334015Z res = mod(**inputs) 2025-11-03T16:41:47.0334266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0334347Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0334594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0334660Z hidden_states = self.encoder( 2025-11-03T16:41:47.0334919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0334984Z layer_outputs = layer_module( 2025-11-03T16:41:47.0335197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0335272Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0335519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0335618Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0335862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0335939Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0336215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0336338Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0336586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0336689Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0336914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0336981Z return self.act(input) 2025-11-03T16:41:47.0336986Z 2025-11-03T16:41:47.0337089Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0337269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0337330Z res = mod(**inputs) 2025-11-03T16:41:47.0337585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0337660Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0337912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0337975Z hidden_states = self.encoder( 2025-11-03T16:41:47.0338242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0338308Z layer_outputs = layer_module( 2025-11-03T16:41:47.0338512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0338595Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0338841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0338938Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0339179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0339249Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0339533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0339658Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0339913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0339990Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0339993Z 2025-11-03T16:41:47.0340093Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0340275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0340335Z res = mod(**inputs) 2025-11-03T16:41:47.0340588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0340663Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0340915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0340983Z hidden_states = self.encoder( 2025-11-03T16:41:47.0341231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0341327Z layer_outputs = layer_module( 2025-11-03T16:41:47.0341532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0341610Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0341857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0341943Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0342190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0342255Z self_outputs = self.self( 2025-11-03T16:41:47.0342529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0342616Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0342621Z 2025-11-03T16:41:47.0342725Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0342908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0342968Z res = mod(**inputs) 2025-11-03T16:41:47.0343230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0343306Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0343561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0343628Z hidden_states = self.encoder( 2025-11-03T16:41:47.0343901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0343967Z layer_outputs = layer_module( 2025-11-03T16:41:47.0344173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0344255Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0344500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0344583Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0344847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0344914Z self_outputs = self.self( 2025-11-03T16:41:47.0345168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0345244Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0345250Z 2025-11-03T16:41:47.0345354Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0345535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0345595Z res = mod(**inputs) 2025-11-03T16:41:47.0345853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0345927Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0346189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0346254Z hidden_states = self.encoder( 2025-11-03T16:41:47.0346505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0346570Z layer_outputs = layer_module( 2025-11-03T16:41:47.0346779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0346858Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0347123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0347207Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0347455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0347523Z self_outputs = self.self( 2025-11-03T16:41:47.0347779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0347866Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0347869Z 2025-11-03T16:41:47.0347952Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0348029Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0348147Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0348333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0348394Z res = mod(**inputs) 2025-11-03T16:41:47.0348651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0348726Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0348992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0349056Z hidden_states = self.encoder( 2025-11-03T16:41:47.0349313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0349385Z layer_outputs = layer_module( 2025-11-03T16:41:47.0349609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0349688Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0349937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0350012Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0350267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0350355Z self_outputs = self.self( 2025-11-03T16:41:47.0350616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0350713Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0350716Z 2025-11-03T16:41:47.0350797Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0350895Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0351077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0351146Z res = mod(**inputs) 2025-11-03T16:41:47.0351405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0351487Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0351736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0351803Z hidden_states = self.encoder( 2025-11-03T16:41:47.0352059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0352123Z layer_outputs = layer_module( 2025-11-03T16:41:47.0352337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0352410Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0352666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0352756Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0353006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0353077Z self_outputs = self.self( 2025-11-03T16:41:47.0353325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0353482Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0353730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0353818Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0353822Z 2025-11-03T16:41:47.0353929Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0354123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0354263Z res = mod(**inputs) 2025-11-03T16:41:47.0354534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0354621Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0354887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0354956Z hidden_states = self.encoder( 2025-11-03T16:41:47.0355225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0355291Z layer_outputs = layer_module( 2025-11-03T16:41:47.0355536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0355615Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0355874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0355956Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0356248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0356344Z self_outputs = self.self( 2025-11-03T16:41:47.0356608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0356772Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0357040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0357110Z x = self.pointwise(x) 2025-11-03T16:41:47.0357115Z 2025-11-03T16:41:47.0357225Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0357419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0357489Z res = mod(**inputs) 2025-11-03T16:41:47.0357752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0357832Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0358100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0358168Z hidden_states = self.encoder( 2025-11-03T16:41:47.0358440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0358508Z layer_outputs = layer_module( 2025-11-03T16:41:47.0358723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0358826Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0359092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0359180Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0359443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0359519Z self_outputs = self.self( 2025-11-03T16:41:47.0359782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0359948Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0359952Z 2025-11-03T16:41:47.0360061Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0360254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0360325Z res = mod(**inputs) 2025-11-03T16:41:47.0360588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0360667Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0360939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0361007Z hidden_states = self.encoder( 2025-11-03T16:41:47.0361278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0361346Z layer_outputs = layer_module( 2025-11-03T16:41:47.0361595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0361673Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0361937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0362025Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0362288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0362384Z self_outputs = self.self( 2025-11-03T16:41:47.0362645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0362761Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0362771Z 2025-11-03T16:41:47.0362874Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0363065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0363138Z res = mod(**inputs) 2025-11-03T16:41:47.0363403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0363489Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0363753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0363823Z hidden_states = self.encoder( 2025-11-03T16:41:47.0364094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0364160Z layer_outputs = layer_module( 2025-11-03T16:41:47.0364385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0364464Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0364726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0364828Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0365087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0365161Z self_outputs = self.self( 2025-11-03T16:41:47.0365424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0365548Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0365551Z 2025-11-03T16:41:47.0365626Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0365698Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0365803Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0366001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0366070Z res = mod(**inputs) 2025-11-03T16:41:47.0366323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0366398Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0366650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0366717Z hidden_states = self.encoder( 2025-11-03T16:41:47.0366972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0367037Z layer_outputs = layer_module( 2025-11-03T16:41:47.0367257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0367336Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0367584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0367670Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0367919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0367987Z self_outputs = self.self( 2025-11-03T16:41:47.0368252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0368359Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0368363Z 2025-11-03T16:41:47.0368471Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0368661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0368730Z res = mod(**inputs) 2025-11-03T16:41:47.0368991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0369071Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0369334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0369400Z hidden_states = self.encoder( 2025-11-03T16:41:47.0369663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0369730Z layer_outputs = layer_module( 2025-11-03T16:41:47.0369947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0370023Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0370280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0370368Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0370637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0370770Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0371025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0371106Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0371117Z 2025-11-03T16:41:47.0371214Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0371399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0371469Z res = mod(**inputs) 2025-11-03T16:41:47.0371743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0371829Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0372093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0372164Z hidden_states = self.encoder( 2025-11-03T16:41:47.0372428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0372500Z layer_outputs = layer_module( 2025-11-03T16:41:47.0372723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0372799Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0373058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0373165Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0373416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0373497Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0373780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0373904Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0374177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0374257Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0374261Z 2025-11-03T16:41:47.0374366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0374554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0374624Z res = mod(**inputs) 2025-11-03T16:41:47.0374875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0374955Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0375221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0375287Z hidden_states = self.encoder( 2025-11-03T16:41:47.0375549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0375615Z layer_outputs = layer_module( 2025-11-03T16:41:47.0375831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0375904Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0376157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0376245Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0376511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0376589Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0376877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0376994Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0377255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0377362Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0377590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0377657Z return self.act(input) 2025-11-03T16:41:47.0377661Z 2025-11-03T16:41:47.0377768Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0377956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0378016Z res = mod(**inputs) 2025-11-03T16:41:47.0378278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0378358Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0378617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0378684Z hidden_states = self.encoder( 2025-11-03T16:41:47.0378951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0379028Z layer_outputs = layer_module( 2025-11-03T16:41:47.0379239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0379320Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0379575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0379654Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0379921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0379995Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0380284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0380413Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0380676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0380757Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0380760Z 2025-11-03T16:41:47.0380857Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0381054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0381114Z res = mod(**inputs) 2025-11-03T16:41:47.0381376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0381452Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0381711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0381779Z hidden_states = self.encoder( 2025-11-03T16:41:47.0382034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0382122Z layer_outputs = layer_module( 2025-11-03T16:41:47.0382338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0382419Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0382687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0382769Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0383043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0383112Z self_outputs = self.self( 2025-11-03T16:41:47.0383380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0383488Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0383492Z 2025-11-03T16:41:47.0383601Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0383795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0383857Z res = mod(**inputs) 2025-11-03T16:41:47.0384135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0384213Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0384480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0384547Z hidden_states = self.encoder( 2025-11-03T16:41:47.0384804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0384901Z layer_outputs = layer_module( 2025-11-03T16:41:47.0385113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0385196Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0385452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0385530Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0385805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0385871Z self_outputs = self.self( 2025-11-03T16:41:47.0386132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0386209Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0386212Z 2025-11-03T16:41:47.0386318Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0386516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0386579Z res = mod(**inputs) 2025-11-03T16:41:47.0386840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0386917Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0387183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0387252Z hidden_states = self.encoder( 2025-11-03T16:41:47.0387508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0387584Z layer_outputs = layer_module( 2025-11-03T16:41:47.0387798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0387877Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0388131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0388233Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0388489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0388558Z self_outputs = self.self( 2025-11-03T16:41:47.0388821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0388910Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0388913Z 2025-11-03T16:41:47.0388995Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0389071Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0389185Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0389384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0389446Z res = mod(**inputs) 2025-11-03T16:41:47.0389709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0389785Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0390039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0390114Z hidden_states = self.encoder( 2025-11-03T16:41:47.0390370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0390443Z layer_outputs = layer_module( 2025-11-03T16:41:47.0390670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0390752Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0391006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0391084Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0391343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0391426Z self_outputs = self.self( 2025-11-03T16:41:47.0391689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0391788Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0391792Z 2025-11-03T16:41:47.0391867Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0391970Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0392164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0392233Z res = mod(**inputs) 2025-11-03T16:41:47.0392488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0392572Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0392826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0392895Z hidden_states = self.encoder( 2025-11-03T16:41:47.0393159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0393226Z layer_outputs = layer_module( 2025-11-03T16:41:47.0393444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0393521Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0393779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0393905Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0394279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0394375Z self_outputs = self.self( 2025-11-03T16:41:47.0394633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0394788Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0395050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0395123Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0395128Z 2025-11-03T16:41:47.0395262Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0395451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0395533Z res = mod(**inputs) 2025-11-03T16:41:47.0395780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0395857Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0396116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0396181Z hidden_states = self.encoder( 2025-11-03T16:41:47.0396435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0396501Z layer_outputs = layer_module( 2025-11-03T16:41:47.0396727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0396806Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0397058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0397141Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0397383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0397469Z self_outputs = self.self( 2025-11-03T16:41:47.0397716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0397862Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0398118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0398183Z x = self.pointwise(x) 2025-11-03T16:41:47.0398186Z 2025-11-03T16:41:47.0398290Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0398474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0398533Z res = mod(**inputs) 2025-11-03T16:41:47.0398786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0398861Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0399114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0399180Z hidden_states = self.encoder( 2025-11-03T16:41:47.0399434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0399502Z layer_outputs = layer_module( 2025-11-03T16:41:47.0399707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0399803Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0400050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0400134Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0400378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0400442Z self_outputs = self.self( 2025-11-03T16:41:47.0400698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0400839Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0400843Z 2025-11-03T16:41:47.0400963Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0401150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0401221Z res = mod(**inputs) 2025-11-03T16:41:47.0401474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0401552Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0401815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0401884Z hidden_states = self.encoder( 2025-11-03T16:41:47.0402144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0402212Z layer_outputs = layer_module( 2025-11-03T16:41:47.0402438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0402521Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0402770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0402854Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0403105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0403191Z self_outputs = self.self( 2025-11-03T16:41:47.0403439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0403549Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0403552Z 2025-11-03T16:41:47.0403655Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0403838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0403904Z res = mod(**inputs) 2025-11-03T16:41:47.0404153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0404226Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0404485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0404552Z hidden_states = self.encoder( 2025-11-03T16:41:47.0404805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0404868Z layer_outputs = layer_module( 2025-11-03T16:41:47.0405078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0405152Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0405399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0405498Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0405751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0405822Z self_outputs = self.self( 2025-11-03T16:41:47.0406075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0406195Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0406199Z 2025-11-03T16:41:47.0406280Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0406354Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0406458Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0406663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0406723Z res = mod(**inputs) 2025-11-03T16:41:47.0406981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0407056Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0407310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0407374Z hidden_states = self.encoder( 2025-11-03T16:41:47.0407632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0407697Z layer_outputs = layer_module( 2025-11-03T16:41:47.0407903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0407997Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0408245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0408330Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0408578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0408642Z self_outputs = self.self( 2025-11-03T16:41:47.0408897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0409017Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0409021Z 2025-11-03T16:41:47.0409124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0409306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0409371Z res = mod(**inputs) 2025-11-03T16:41:47.0409620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0409696Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0409949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0410013Z hidden_states = self.encoder( 2025-11-03T16:41:47.0410263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0410329Z layer_outputs = layer_module( 2025-11-03T16:41:47.0410531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0410608Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0410852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0410931Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0411176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0411319Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0411565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0411643Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0411646Z 2025-11-03T16:41:47.0411749Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0411928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0411994Z res = mod(**inputs) 2025-11-03T16:41:47.0412256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0412333Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0412586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0412653Z hidden_states = self.encoder( 2025-11-03T16:41:47.0412911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0412976Z layer_outputs = layer_module( 2025-11-03T16:41:47.0413187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0413361Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0413618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0413725Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0413971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0414052Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0414337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0414452Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0414725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0414803Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0414807Z 2025-11-03T16:41:47.0414910Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0415095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0415166Z res = mod(**inputs) 2025-11-03T16:41:47.0415419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0415496Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0415751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0415818Z hidden_states = self.encoder( 2025-11-03T16:41:47.0416073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0416142Z layer_outputs = layer_module( 2025-11-03T16:41:47.0416347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0416426Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0416673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0416757Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0417040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0417111Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0417392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0417506Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0417758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0417861Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0418065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0418155Z return self.act(input) 2025-11-03T16:41:47.0418159Z 2025-11-03T16:41:47.0418255Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0418448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0418508Z res = mod(**inputs) 2025-11-03T16:41:47.0418771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0418848Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0419096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0419173Z hidden_states = self.encoder( 2025-11-03T16:41:47.0419425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0419522Z layer_outputs = layer_module( 2025-11-03T16:41:47.0419734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0419814Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0420063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0420140Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0420390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0420482Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0420768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0420893Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0421147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0421232Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0421237Z 2025-11-03T16:41:47.0421334Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0421528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0421587Z res = mod(**inputs) 2025-11-03T16:41:47.0421845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0421921Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0422170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0422245Z hidden_states = self.encoder( 2025-11-03T16:41:47.0422495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0422566Z layer_outputs = layer_module( 2025-11-03T16:41:47.0422786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0422859Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0423111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0423186Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0423439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0423503Z self_outputs = self.self( 2025-11-03T16:41:47.0423754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0423856Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0423859Z 2025-11-03T16:41:47.0423954Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0424147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0424207Z res = mod(**inputs) 2025-11-03T16:41:47.0424463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0424537Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0424785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0424858Z hidden_states = self.encoder( 2025-11-03T16:41:47.0425104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0425177Z layer_outputs = layer_module( 2025-11-03T16:41:47.0425398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0425479Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0425732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0425807Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0426061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0426141Z self_outputs = self.self( 2025-11-03T16:41:47.0426394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0426471Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0426474Z 2025-11-03T16:41:47.0426568Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0426762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0426822Z res = mod(**inputs) 2025-11-03T16:41:47.0427075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0427150Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0427401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0427468Z hidden_states = self.encoder( 2025-11-03T16:41:47.0427714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0427786Z layer_outputs = layer_module( 2025-11-03T16:41:47.0427989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0428069Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0428312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0428410Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0428667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0428732Z self_outputs = self.self( 2025-11-03T16:41:47.0428998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0429083Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0429086Z 2025-11-03T16:41:47.0429170Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0429245Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0429341Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0429549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0429609Z res = mod(**inputs) 2025-11-03T16:41:47.0429868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0429943Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0430192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0430269Z hidden_states = self.encoder( 2025-11-03T16:41:47.0430526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0430598Z layer_outputs = layer_module( 2025-11-03T16:41:47.0430814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0430906Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0431170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0431248Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0431511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0431576Z self_outputs = self.self( 2025-11-03T16:41:47.0431853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0431951Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0431954Z 2025-11-03T16:41:47.0432038Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0432144Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0432333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0432402Z res = mod(**inputs) 2025-11-03T16:41:47.0432654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0432732Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0432990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0433059Z hidden_states = self.encoder( 2025-11-03T16:41:47.0433319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0433385Z layer_outputs = layer_module( 2025-11-03T16:41:47.0433593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0433674Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0433931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0434033Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0434349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0434427Z self_outputs = self.self( 2025-11-03T16:41:47.0434685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0434845Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0435115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0435189Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0435193Z 2025-11-03T16:41:47.0435333Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0435524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0435589Z res = mod(**inputs) 2025-11-03T16:41:47.0435853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0435931Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0436199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0436269Z hidden_states = self.encoder( 2025-11-03T16:41:47.0436541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0436608Z layer_outputs = layer_module( 2025-11-03T16:41:47.0436843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0436929Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0437208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0437295Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0437572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0437639Z self_outputs = self.self( 2025-11-03T16:41:47.0437939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0438110Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0438386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0438456Z x = self.pointwise(x) 2025-11-03T16:41:47.0438461Z 2025-11-03T16:41:47.0438568Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0438759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0438820Z res = mod(**inputs) 2025-11-03T16:41:47.0439115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0439193Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0439475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0439543Z hidden_states = self.encoder( 2025-11-03T16:41:47.0455250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0455403Z layer_outputs = layer_module( 2025-11-03T16:41:47.0455683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0455776Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0456155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0456248Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0456507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0456592Z self_outputs = self.self( 2025-11-03T16:41:47.0456847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0456997Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0457006Z 2025-11-03T16:41:47.0457126Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0457358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0457435Z res = mod(**inputs) 2025-11-03T16:41:47.0457693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0457774Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0458029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0458102Z hidden_states = self.encoder( 2025-11-03T16:41:47.0458358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0458427Z layer_outputs = layer_module( 2025-11-03T16:41:47.0458674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0458755Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0459009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0459097Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0459347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0459423Z self_outputs = self.self( 2025-11-03T16:41:47.0459704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0459820Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0459825Z 2025-11-03T16:41:47.0459935Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0460130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0460205Z res = mod(**inputs) 2025-11-03T16:41:47.0460459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0460549Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0460797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0460866Z hidden_states = self.encoder( 2025-11-03T16:41:47.0461122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0461190Z layer_outputs = layer_module( 2025-11-03T16:41:47.0461408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0461483Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0461732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0461817Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0462082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0462157Z self_outputs = self.self( 2025-11-03T16:41:47.0462407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0462531Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0462544Z 2025-11-03T16:41:47.0462625Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0462699Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0462805Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0462994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0463078Z res = mod(**inputs) 2025-11-03T16:41:47.0463328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0463407Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0463661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0463728Z hidden_states = self.encoder( 2025-11-03T16:41:47.0463984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0464050Z layer_outputs = layer_module( 2025-11-03T16:41:47.0464256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0464338Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0464624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0464710Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0464958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0465030Z self_outputs = self.self( 2025-11-03T16:41:47.0465277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0465400Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0465404Z 2025-11-03T16:41:47.0465510Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0465700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0465767Z res = mod(**inputs) 2025-11-03T16:41:47.0466022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0466099Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0466359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0466427Z hidden_states = self.encoder( 2025-11-03T16:41:47.0466686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0466752Z layer_outputs = layer_module( 2025-11-03T16:41:47.0466962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0467041Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0467289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0467378Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0467630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0467779Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0468029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0468110Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0468114Z 2025-11-03T16:41:47.0468221Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0468404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0468472Z res = mod(**inputs) 2025-11-03T16:41:47.0468721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0468814Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0469071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0469137Z hidden_states = self.encoder( 2025-11-03T16:41:47.0469393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0469459Z layer_outputs = layer_module( 2025-11-03T16:41:47.0469673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0469746Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0469992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0470079Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0470339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0470420Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0470702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0470816Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0471073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0471167Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0471170Z 2025-11-03T16:41:47.0471274Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0471457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0471523Z res = mod(**inputs) 2025-11-03T16:41:47.0471774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0471849Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0472106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0472171Z hidden_states = self.encoder( 2025-11-03T16:41:47.0472425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0472492Z layer_outputs = layer_module( 2025-11-03T16:41:47.0472699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0472779Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0473028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0473117Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0473362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0473463Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0473742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0473857Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0474128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0474359Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0474592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0474664Z return self.act(input) 2025-11-03T16:41:47.0474671Z 2025-11-03T16:41:47.0474795Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0475000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0475065Z res = mod(**inputs) 2025-11-03T16:41:47.0475334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0475416Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0475687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0475758Z hidden_states = self.encoder( 2025-11-03T16:41:47.0476068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0476147Z layer_outputs = layer_module( 2025-11-03T16:41:47.0476381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0476469Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0476732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0476813Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0477081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0477175Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0477474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0477607Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0477884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0477967Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0477971Z 2025-11-03T16:41:47.0478078Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0478285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0478353Z res = mod(**inputs) 2025-11-03T16:41:47.0478628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0478712Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0478979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0479061Z hidden_states = self.encoder( 2025-11-03T16:41:47.0479328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0479409Z layer_outputs = layer_module( 2025-11-03T16:41:47.0479632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0479737Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0480003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0480084Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0480363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0480434Z self_outputs = self.self( 2025-11-03T16:41:47.0480706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0480800Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0480804Z 2025-11-03T16:41:47.0480922Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0481129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0481196Z res = mod(**inputs) 2025-11-03T16:41:47.0481468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0481547Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0481819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0481890Z hidden_states = self.encoder( 2025-11-03T16:41:47.0482156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0482236Z layer_outputs = layer_module( 2025-11-03T16:41:47.0482479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0482566Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0482847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0482927Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0483200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0483291Z self_outputs = self.self( 2025-11-03T16:41:47.0483566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0483649Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0483652Z 2025-11-03T16:41:47.0483761Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0483959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0484023Z res = mod(**inputs) 2025-11-03T16:41:47.0484293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0484376Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0484646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0484714Z hidden_states = self.encoder( 2025-11-03T16:41:47.0484977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0485053Z layer_outputs = layer_module( 2025-11-03T16:41:47.0485273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0485357Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0485622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0485719Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0485996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0486064Z self_outputs = self.self( 2025-11-03T16:41:47.0486332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0486427Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0486430Z 2025-11-03T16:41:47.0486518Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0486596Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0486697Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0486921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0486984Z res = mod(**inputs) 2025-11-03T16:41:47.0487251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0487332Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0487591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0487667Z hidden_states = self.encoder( 2025-11-03T16:41:47.0487933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0488009Z layer_outputs = layer_module( 2025-11-03T16:41:47.0488225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0488300Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0488587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0488668Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0488938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0489006Z self_outputs = self.self( 2025-11-03T16:41:47.0489279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0489397Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0489401Z 2025-11-03T16:41:47.0489480Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0489591Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0489785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0489857Z res = mod(**inputs) 2025-11-03T16:41:47.0490125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0490208Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0490481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0490549Z hidden_states = self.encoder( 2025-11-03T16:41:47.0490821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0490901Z layer_outputs = layer_module( 2025-11-03T16:41:47.0491115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0491186Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0491433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0491516Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0491777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0491850Z self_outputs = self.self( 2025-11-03T16:41:47.0492098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0492255Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0492510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0492583Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0492586Z 2025-11-03T16:41:47.0492692Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0492889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0492957Z res = mod(**inputs) 2025-11-03T16:41:47.0493206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0493282Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0493537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0493603Z hidden_states = self.encoder( 2025-11-03T16:41:47.0493858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0493923Z layer_outputs = layer_module( 2025-11-03T16:41:47.0494130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0494227Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0494471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0494557Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0494804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0494876Z self_outputs = self.self( 2025-11-03T16:41:47.0495123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0495290Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0495546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0495615Z x = self.pointwise(x) 2025-11-03T16:41:47.0495619Z 2025-11-03T16:41:47.0495727Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0495909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0495979Z res = mod(**inputs) 2025-11-03T16:41:47.0496227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0496304Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0496563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0496629Z hidden_states = self.encoder( 2025-11-03T16:41:47.0496885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0496951Z layer_outputs = layer_module( 2025-11-03T16:41:47.0497161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0497240Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0497503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0497588Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0497836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0497911Z self_outputs = self.self( 2025-11-03T16:41:47.0498155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0498297Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0498301Z 2025-11-03T16:41:47.0498404Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0498603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0498672Z res = mod(**inputs) 2025-11-03T16:41:47.0498920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0498994Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0499261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0499329Z hidden_states = self.encoder( 2025-11-03T16:41:47.0499596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0499660Z layer_outputs = layer_module( 2025-11-03T16:41:47.0499875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0499994Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0500246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0500329Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0500576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0500645Z self_outputs = self.self( 2025-11-03T16:41:47.0500893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0501019Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0501029Z 2025-11-03T16:41:47.0501124Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0501308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0501378Z res = mod(**inputs) 2025-11-03T16:41:47.0501631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0501712Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0501962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0502026Z hidden_states = self.encoder( 2025-11-03T16:41:47.0502285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0502349Z layer_outputs = layer_module( 2025-11-03T16:41:47.0502564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0502634Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0502898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0502972Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0503228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0503308Z self_outputs = self.self( 2025-11-03T16:41:47.0503554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0503679Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0503682Z 2025-11-03T16:41:47.0503756Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0503836Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0503930Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0504112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0504180Z res = mod(**inputs) 2025-11-03T16:41:47.0504445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0504529Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0504776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0504841Z hidden_states = self.encoder( 2025-11-03T16:41:47.0505093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0505158Z layer_outputs = layer_module( 2025-11-03T16:41:47.0505368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0505440Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0505709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0505784Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0506046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0506116Z self_outputs = self.self( 2025-11-03T16:41:47.0506365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0506494Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0506498Z 2025-11-03T16:41:47.0506593Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0506776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0506843Z res = mod(**inputs) 2025-11-03T16:41:47.0507092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0507173Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0507419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0507492Z hidden_states = self.encoder( 2025-11-03T16:41:47.0507740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0507806Z layer_outputs = layer_module( 2025-11-03T16:41:47.0508017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0508087Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0508335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0508408Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0508653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0508805Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0509056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0509140Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0509143Z 2025-11-03T16:41:47.0509242Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0509436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0509496Z res = mod(**inputs) 2025-11-03T16:41:47.0509759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0509843Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0510119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0510194Z hidden_states = self.encoder( 2025-11-03T16:41:47.0510457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0510523Z layer_outputs = layer_module( 2025-11-03T16:41:47.0510740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0510816Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0511078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0511158Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0511428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0511502Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0511787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0511909Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0512164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0512264Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0512268Z 2025-11-03T16:41:47.0512366Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0512554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0512620Z res = mod(**inputs) 2025-11-03T16:41:47.0512884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0512967Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0513393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0513511Z hidden_states = self.encoder( 2025-11-03T16:41:47.0513845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0513915Z layer_outputs = layer_module( 2025-11-03T16:41:47.0514136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0514313Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0514591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0514679Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0514946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0515098Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0515400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0515524Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0515778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0515893Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0516101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0516171Z return self.act(input) 2025-11-03T16:41:47.0516174Z 2025-11-03T16:41:47.0516310Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0516502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0516573Z res = mod(**inputs) 2025-11-03T16:41:47.0516833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0516913Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0517177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0517246Z hidden_states = self.encoder( 2025-11-03T16:41:47.0517511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0517578Z layer_outputs = layer_module( 2025-11-03T16:41:47.0517814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0517895Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0518149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0518237Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0518490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0518604Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0518895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0519024Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0519287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0519368Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0519371Z 2025-11-03T16:41:47.0519478Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0519671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0519732Z res = mod(**inputs) 2025-11-03T16:41:47.0519998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0520075Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0520337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0520404Z hidden_states = self.encoder( 2025-11-03T16:41:47.0520666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0520733Z layer_outputs = layer_module( 2025-11-03T16:41:47.0520947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0521042Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0521297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0521379Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0521634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0521703Z self_outputs = self.self( 2025-11-03T16:41:47.0521966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0522054Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0522058Z 2025-11-03T16:41:47.0522165Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0522369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0522440Z res = mod(**inputs) 2025-11-03T16:41:47.0522701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0522778Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0523038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0523104Z hidden_states = self.encoder( 2025-11-03T16:41:47.0523364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0523430Z layer_outputs = layer_module( 2025-11-03T16:41:47.0523653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0523735Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0523989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0524072Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0524325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0524398Z self_outputs = self.self( 2025-11-03T16:41:47.0524666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0524743Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0524746Z 2025-11-03T16:41:47.0524851Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0525043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0525112Z res = mod(**inputs) 2025-11-03T16:41:47.0525370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0525447Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0525716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0525779Z hidden_states = self.encoder( 2025-11-03T16:41:47.0526034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0526098Z layer_outputs = layer_module( 2025-11-03T16:41:47.0526308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0526380Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0526629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0526710Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0526972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0527042Z self_outputs = self.self( 2025-11-03T16:41:47.0527296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0527383Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0527386Z 2025-11-03T16:41:47.0527467Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0527542Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0527644Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0527827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0527903Z res = mod(**inputs) 2025-11-03T16:41:47.0528157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0528237Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0528491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0528556Z hidden_states = self.encoder( 2025-11-03T16:41:47.0528818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0528882Z layer_outputs = layer_module( 2025-11-03T16:41:47.0529087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0529164Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0529426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0529509Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0529760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0529824Z self_outputs = self.self( 2025-11-03T16:41:47.0530080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0530200Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0530203Z 2025-11-03T16:41:47.0530286Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0530381Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0530568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0530628Z res = mod(**inputs) 2025-11-03T16:41:47.0530880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0530961Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0531211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0531284Z hidden_states = self.encoder( 2025-11-03T16:41:47.0531531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0531596Z layer_outputs = layer_module( 2025-11-03T16:41:47.0531808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0531879Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0532145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0532222Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0532468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0532557Z self_outputs = self.self( 2025-11-03T16:41:47.0532810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0532968Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0533218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0533298Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0533301Z 2025-11-03T16:41:47.0533397Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0533593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0533661Z res = mod(**inputs) 2025-11-03T16:41:47.0533910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0533994Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0534243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0534307Z hidden_states = self.encoder( 2025-11-03T16:41:47.0534566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0534631Z layer_outputs = layer_module( 2025-11-03T16:41:47.0534844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0534916Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0535186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0535264Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0535508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0535581Z self_outputs = self.self( 2025-11-03T16:41:47.0535830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0535999Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0536246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0536309Z x = self.pointwise(x) 2025-11-03T16:41:47.0536321Z 2025-11-03T16:41:47.0536418Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0536602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0536669Z res = mod(**inputs) 2025-11-03T16:41:47.0536917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0536996Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0537242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0537307Z hidden_states = self.encoder( 2025-11-03T16:41:47.0537560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0537622Z layer_outputs = layer_module( 2025-11-03T16:41:47.0537834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0537907Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0538152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0538247Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0538496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0538565Z self_outputs = self.self( 2025-11-03T16:41:47.0538812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0538958Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0538962Z 2025-11-03T16:41:47.0539056Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0539253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0539320Z res = mod(**inputs) 2025-11-03T16:41:47.0539569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0539650Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0539895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0539959Z hidden_states = self.encoder( 2025-11-03T16:41:47.0540216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0540282Z layer_outputs = layer_module( 2025-11-03T16:41:47.0540492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0540561Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0540830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0540904Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0541151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0541222Z self_outputs = self.self( 2025-11-03T16:41:47.0541468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0541598Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0541602Z 2025-11-03T16:41:47.0541697Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0541876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0541942Z res = mod(**inputs) 2025-11-03T16:41:47.0542192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0542273Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0542522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0542595Z hidden_states = self.encoder( 2025-11-03T16:41:47.0542844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0542909Z layer_outputs = layer_module( 2025-11-03T16:41:47.0543120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0543190Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0543444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0543521Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0543766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0543855Z self_outputs = self.self( 2025-11-03T16:41:47.0544102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0544228Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0544232Z 2025-11-03T16:41:47.0544306Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0544379Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0544482Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0544663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0544729Z res = mod(**inputs) 2025-11-03T16:41:47.0544995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0545077Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0545325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0545390Z hidden_states = self.encoder( 2025-11-03T16:41:47.0545643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0545710Z layer_outputs = layer_module( 2025-11-03T16:41:47.0545921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0545991Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0546250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0546334Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0546581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0546653Z self_outputs = self.self( 2025-11-03T16:41:47.0546902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0547014Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0547031Z 2025-11-03T16:41:47.0547127Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0547310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0547376Z res = mod(**inputs) 2025-11-03T16:41:47.0547625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0547709Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0547956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0548023Z hidden_states = self.encoder( 2025-11-03T16:41:47.0548277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0548340Z layer_outputs = layer_module( 2025-11-03T16:41:47.0548554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0548624Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0548872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0548952Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0549202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0549330Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0549594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0549676Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0549680Z 2025-11-03T16:41:47.0549772Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0549955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0550021Z res = mod(**inputs) 2025-11-03T16:41:47.0550271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0550348Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0550614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0550688Z hidden_states = self.encoder( 2025-11-03T16:41:47.0550938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0551001Z layer_outputs = layer_module( 2025-11-03T16:41:47.0551214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0551286Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0551537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0551613Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0551855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0551954Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0552232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0552354Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0552601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0552676Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0552701Z 2025-11-03T16:41:47.0552797Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0552979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0553044Z res = mod(**inputs) 2025-11-03T16:41:47.0553290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0553374Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0553620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0553688Z hidden_states = self.encoder( 2025-11-03T16:41:47.0553998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0554066Z layer_outputs = layer_module( 2025-11-03T16:41:47.0554399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0554480Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0554744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0554832Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0555092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0555174Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0555495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0555629Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0555879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0555988Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0556198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0556265Z return self.act(input) 2025-11-03T16:41:47.0556268Z 2025-11-03T16:41:47.0556374Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0556573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0556634Z res = mod(**inputs) 2025-11-03T16:41:47.0556893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0556968Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0557225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0557293Z hidden_states = self.encoder( 2025-11-03T16:41:47.0557548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0557613Z layer_outputs = layer_module( 2025-11-03T16:41:47.0557822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0557919Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0558166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0558250Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0558489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0558558Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0558856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0558979Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0559240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0559317Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0559321Z 2025-11-03T16:41:47.0559426Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0559617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0559679Z res = mod(**inputs) 2025-11-03T16:41:47.0559941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0560017Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0560288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0560356Z hidden_states = self.encoder( 2025-11-03T16:41:47.0560611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0560685Z layer_outputs = layer_module( 2025-11-03T16:41:47.0560901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0560981Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0561252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0561336Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0561591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0561660Z self_outputs = self.self( 2025-11-03T16:41:47.0561923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-11-03T16:41:47.0562010Z mixed_query_layer = self.query(hidden_states) 2025-11-03T16:41:47.0562013Z 2025-11-03T16:41:47.0562116Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0562321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0562382Z res = mod(**inputs) 2025-11-03T16:41:47.0562650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0562726Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0562987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0563056Z hidden_states = self.encoder( 2025-11-03T16:41:47.0563315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0563382Z layer_outputs = layer_module( 2025-11-03T16:41:47.0563594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0563690Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0563947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0564032Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0564288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0564354Z self_outputs = self.self( 2025-11-03T16:41:47.0564617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-11-03T16:41:47.0564712Z mixed_key_layer = self.key(hidden_states) 2025-11-03T16:41:47.0564716Z 2025-11-03T16:41:47.0564820Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0565005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0565072Z res = mod(**inputs) 2025-11-03T16:41:47.0565333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0565411Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0565674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0565739Z hidden_states = self.encoder( 2025-11-03T16:41:47.0566001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0566067Z layer_outputs = layer_module( 2025-11-03T16:41:47.0566279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0566358Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0566619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0566702Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0566958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0567041Z self_outputs = self.self( 2025-11-03T16:41:47.0567307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-11-03T16:41:47.0567393Z mixed_value_layer = self.value(hidden_states) 2025-11-03T16:41:47.0567398Z 2025-11-03T16:41:47.0567478Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0567552Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0567658Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0567844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0567903Z res = mod(**inputs) 2025-11-03T16:41:47.0568182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0568260Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0568524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0568590Z hidden_states = self.encoder( 2025-11-03T16:41:47.0568847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0568922Z layer_outputs = layer_module( 2025-11-03T16:41:47.0569134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0569214Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0569491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0569568Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0569828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0569897Z self_outputs = self.self( 2025-11-03T16:41:47.0570159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-11-03T16:41:47.0570258Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-11-03T16:41:47.0570276Z 2025-11-03T16:41:47.0570359Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0570457Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0570644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0570714Z res = mod(**inputs) 2025-11-03T16:41:47.0570976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0571060Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0571323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0571391Z hidden_states = self.encoder( 2025-11-03T16:41:47.0571657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0571724Z layer_outputs = layer_module( 2025-11-03T16:41:47.0571943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0572018Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0572279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0572359Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0572620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0572710Z self_outputs = self.self( 2025-11-03T16:41:47.0572974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0573133Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0573398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-11-03T16:41:47.0573471Z x = self.depthwise(hidden_states) 2025-11-03T16:41:47.0573481Z 2025-11-03T16:41:47.0573582Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0573770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0573839Z res = mod(**inputs) 2025-11-03T16:41:47.0574131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0574216Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0574473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0574540Z hidden_states = self.encoder( 2025-11-03T16:41:47.0574806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0574874Z layer_outputs = layer_module( 2025-11-03T16:41:47.0575094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0575167Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0575445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0575530Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0575780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0575852Z self_outputs = self.self( 2025-11-03T16:41:47.0576105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-11-03T16:41:47.0576277Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-11-03T16:41:47.0576533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-11-03T16:41:47.0576600Z x = self.pointwise(x) 2025-11-03T16:41:47.0576603Z 2025-11-03T16:41:47.0576709Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0576898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0576964Z res = mod(**inputs) 2025-11-03T16:41:47.0577229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0577304Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0577570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0577637Z hidden_states = self.encoder( 2025-11-03T16:41:47.0577900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0577965Z layer_outputs = layer_module( 2025-11-03T16:41:47.0578183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0578257Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0578514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0578615Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0578869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0578941Z self_outputs = self.self( 2025-11-03T16:41:47.0579194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-11-03T16:41:47.0579340Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-11-03T16:41:47.0579344Z 2025-11-03T16:41:47.0579448Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0579636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0579704Z res = mod(**inputs) 2025-11-03T16:41:47.0579976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0580058Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0580316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0580381Z hidden_states = self.encoder( 2025-11-03T16:41:47.0580641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0580709Z layer_outputs = layer_module( 2025-11-03T16:41:47.0580924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0580996Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0581261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0581343Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0581600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0581670Z self_outputs = self.self( 2025-11-03T16:41:47.0581924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-11-03T16:41:47.0582050Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-11-03T16:41:47.0582059Z 2025-11-03T16:41:47.0582159Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0582353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0582421Z res = mod(**inputs) 2025-11-03T16:41:47.0582683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0582773Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0583032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0583101Z hidden_states = self.encoder( 2025-11-03T16:41:47.0583369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0583437Z layer_outputs = layer_module( 2025-11-03T16:41:47.0583657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0583732Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0583990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0584085Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0584335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0584424Z self_outputs = self.self( 2025-11-03T16:41:47.0584690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-11-03T16:41:47.0584820Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-11-03T16:41:47.0584823Z 2025-11-03T16:41:47.0584903Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0584979Z cudagraph partition due to non gpu ops 2025-11-03T16:41:47.0585087Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0585282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0585349Z res = mod(**inputs) 2025-11-03T16:41:47.0585643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0585723Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0585993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0586071Z hidden_states = self.encoder( 2025-11-03T16:41:47.0586340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0586410Z layer_outputs = layer_module( 2025-11-03T16:41:47.0586627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0586709Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0586970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0587072Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0587338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-11-03T16:41:47.0587415Z self_outputs = self.self( 2025-11-03T16:41:47.0587679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-11-03T16:41:47.0587790Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-11-03T16:41:47.0587793Z 2025-11-03T16:41:47.0587916Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0588108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0588176Z res = mod(**inputs) 2025-11-03T16:41:47.0588446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0588524Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0588796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0588866Z hidden_states = self.encoder( 2025-11-03T16:41:47.0589131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0589198Z layer_outputs = layer_module( 2025-11-03T16:41:47.0589421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0589497Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0589759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-11-03T16:41:47.0589843Z self_attention_outputs = self.attention( 2025-11-03T16:41:47.0590106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-11-03T16:41:47.0590238Z attention_output = self.output(self_outputs[0], hidden_states) 2025-11-03T16:41:47.0590515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-11-03T16:41:47.0590596Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0590607Z 2025-11-03T16:41:47.0590708Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0590904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0590975Z res = mod(**inputs) 2025-11-03T16:41:47.0591237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0591322Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0591632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0591703Z hidden_states = self.encoder( 2025-11-03T16:41:47.0591975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0592042Z layer_outputs = layer_module( 2025-11-03T16:41:47.0592254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0592325Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0592572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0592657Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0592895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0592973Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0593267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0593392Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0593637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-11-03T16:41:47.0593713Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0593717Z 2025-11-03T16:41:47.0593838Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0594021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0594087Z res = mod(**inputs) 2025-11-03T16:41:47.0594440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0594522Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0594794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0594863Z hidden_states = self.encoder( 2025-11-03T16:41:47.0595129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0595195Z layer_outputs = layer_module( 2025-11-03T16:41:47.0595417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0595491Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0595796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0595882Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0596127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0596205Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0596486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-11-03T16:41:47.0596630Z intermediate_output = self.intermediate(attention_output) 2025-11-03T16:41:47.0596884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-11-03T16:41:47.0596990Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-11-03T16:41:47.0597197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 70, in forward 2025-11-03T16:41:47.0597262Z return self.act(input) 2025-11-03T16:41:47.0597266Z 2025-11-03T16:41:47.0597369Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0597570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0597632Z res = mod(**inputs) 2025-11-03T16:41:47.0597891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-11-03T16:41:47.0597969Z generator_hidden_states = self.convbert( 2025-11-03T16:41:47.0598228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-11-03T16:41:47.0598293Z hidden_states = self.encoder( 2025-11-03T16:41:47.0598546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-11-03T16:41:47.0598618Z layer_outputs = layer_module( 2025-11-03T16:41:47.0598828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-11-03T16:41:47.0598906Z return super().__call__(*args, **kwargs) 2025-11-03T16:41:47.0599181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-11-03T16:41:47.0599268Z layer_output = apply_chunking_to_forward( 2025-11-03T16:41:47.0599507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-11-03T16:41:47.0599578Z return forward_fn(*input_tensors) 2025-11-03T16:41:47.0599863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-11-03T16:41:47.0600001Z layer_output = self.output(intermediate_output, attention_output) 2025-11-03T16:41:47.0600259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-11-03T16:41:47.0600336Z hidden_states = self.dense(hidden_states) 2025-11-03T16:41:47.0600339Z 2025-11-03T16:41:47.0600439Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0600631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0600693Z res = mod(**inputs) 2025-11-03T16:41:47.0600949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-11-03T16:41:47.0601092Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-11-03T16:41:47.0601347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 876, in forward 2025-11-03T16:41:47.0601445Z hidden_states = self.dense(generator_hidden_states) 2025-11-03T16:41:47.0601448Z 2025-11-03T16:41:47.0601544Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0601736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0601797Z res = mod(**inputs) 2025-11-03T16:41:47.0602059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 939, in forward 2025-11-03T16:41:47.0602198Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-11-03T16:41:47.0602201Z 2025-11-03T16:41:47.0602295Z cudagraph partition due to non gpu ops. Found from : 2025-11-03T16:41:47.0602483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-11-03T16:41:47.0602543Z res = mod(**inputs) 2025-11-03T16:41:47.0602797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 945, in forward 2025-11-03T16:41:47.0602954Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-11-03T16:41:47.0602958Z 2025-11-03T16:41:57.5948831Z Compilation time (from dynamo_timed): 21.287408556 2025-11-03T16:41:57.5987414Z pass 2025-11-03T16:41:57.5988158Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-11-03T16:41:57.5989080Z TIMING: _recursive_pre_grad_passes:0.00913 _recursive_joint_graph_passes:0.59003 _recursive_post_grad_passes:0.15375 async_compile.wait:0.63262 code_gen:10.0337 inductor_compile:12.44436 backend_compile:17.22505 gc:0.00052 entire_frame_compile:21.28741 total_wall_time:21.28741 2025-11-03T16:41:57.5990108Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:13492 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:3940 2025-11-03T16:41:57.5990601Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-11-03T16:41:59.3012061Z accuracy pass_rate=92.59% 2025-11-03T16:41:59.3012862Z calls_captured gmean=0.00x mean=579.519x 2025-11-03T16:41:59.3013119Z unique_graphs gmean=0.00x mean=1.111x 2025-11-03T16:41:59.3013480Z graph_breaks gmean=0.00x mean=0.222x 2025-11-03T16:41:59.3019364Z unique_graph_breaks gmean=0.00x mean=0.074x 2025-11-03T16:41:59.3023917Z autograd_captures gmean=0.00x mean=0.000x 2025-11-03T16:41:59.3028388Z autograd_compiles gmean=0.00x mean=0.000x 2025-11-03T16:41:59.3028841Z cudagraph_skips gmean=0.00x mean=1.111x 2025-11-03T16:41:59.3029180Z compilation_latency mean=20.097 seconds 2025-11-03T16:42:00.1848232Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-11-03T16:42:00.4536939Z AlbertForMaskedLM PASS 2025-11-03T16:42:00.4537473Z AllenaiLongformerBase PASS 2025-11-03T16:42:00.4541686Z BartForCausalLM PASS 2025-11-03T16:42:00.4543774Z BertForMaskedLM PASS 2025-11-03T16:42:00.4549104Z BlenderbotForCausalLM XFAIL 2025-11-03T16:42:00.4553714Z DebertaV2ForMaskedLM XFAIL 2025-11-03T16:42:00.4554049Z DistilBertForMaskedLM PASS 2025-11-03T16:42:00.4559427Z DistillGPT2 PASS 2025-11-03T16:42:00.4561600Z ElectraForCausalLM PASS 2025-11-03T16:42:00.4561997Z GPT2ForSequenceClassification PASS 2025-11-03T16:42:00.4567375Z GoogleFnet PASS 2025-11-03T16:42:00.4567844Z LayoutLMForMaskedLM PASS 2025-11-03T16:42:00.4574581Z M2M100ForConditionalGeneration PASS 2025-11-03T16:42:00.4576727Z MBartForCausalLM PASS 2025-11-03T16:42:00.4577152Z MT5ForConditionalGeneration PASS 2025-11-03T16:42:00.4582666Z MegatronBertForCausalLM PASS 2025-11-03T16:42:00.4584767Z MobileBertForMaskedLM PASS 2025-11-03T16:42:00.4585163Z OPTForCausalLM PASS 2025-11-03T16:42:00.4587541Z PLBartForCausalLM PASS 2025-11-03T16:42:00.4587962Z PegasusForCausalLM PASS 2025-11-03T16:42:00.4594914Z RobertaForCausalLM PASS 2025-11-03T16:42:00.4600276Z T5ForConditionalGeneration PASS 2025-11-03T16:42:00.4602329Z T5Small PASS 2025-11-03T16:42:00.4602679Z TrOCRForCausalLM PASS 2025-11-03T16:42:00.4608033Z XGLMForCausalLM PASS 2025-11-03T16:42:00.4610111Z XLNetLMHeadModel PASS 2025-11-03T16:42:00.4610380Z YituTechConvBert PASS 2025-11-03T16:42:00.5088860Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-11-03T16:42:00.7676397Z AlbertForMaskedLM PASS 2025-11-03T16:42:00.7679187Z AllenaiLongformerBase PASS 2025-11-03T16:42:00.7684254Z BartForCausalLM PASS 2025-11-03T16:42:00.7689642Z BertForMaskedLM PASS 2025-11-03T16:42:00.7691421Z BlenderbotForCausalLM PASS 2025-11-03T16:42:00.7695234Z DebertaV2ForMaskedLM PASS 2025-11-03T16:42:00.7700516Z DistilBertForMaskedLM PASS 2025-11-03T16:42:00.7700903Z DistillGPT2 PASS 2025-11-03T16:42:00.7701217Z ElectraForCausalLM PASS 2025-11-03T16:42:00.7701447Z GPT2ForSequenceClassification PASS 2025-11-03T16:42:00.7707452Z GoogleFnet PASS 2025-11-03T16:42:00.7711893Z LayoutLMForMaskedLM PASS 2025-11-03T16:42:00.7714334Z M2M100ForConditionalGeneration PASS 2025-11-03T16:42:00.7717787Z MBartForCausalLM PASS 2025-11-03T16:42:00.7718149Z MT5ForConditionalGeneration PASS 2025-11-03T16:42:00.7722511Z MegatronBertForCausalLM PASS 2025-11-03T16:42:00.7727046Z MobileBertForMaskedLM PASS 2025-11-03T16:42:00.7732441Z OPTForCausalLM PASS 2025-11-03T16:42:00.7734373Z PLBartForCausalLM PASS 2025-11-03T16:42:00.7734923Z PegasusForCausalLM PASS 2025-11-03T16:42:00.7739828Z RobertaForCausalLM PASS 2025-11-03T16:42:00.7741893Z T5ForConditionalGeneration PASS 2025-11-03T16:42:00.7742253Z T5Small PASS 2025-11-03T16:42:00.7746859Z TrOCRForCausalLM PASS 2025-11-03T16:42:00.7749249Z XGLMForCausalLM PASS_BUT_FLAKY 2025-11-03T16:42:00.7753499Z XLNetLMHeadModel PASS 2025-11-03T16:42:00.7757279Z YituTechConvBert PASS 2025-11-03T16:42:00.8291677Z + sccache_epilogue 2025-11-03T16:42:00.8296439Z + echo '::group::Sccache Compilation Log' 2025-11-03T16:42:00.8298115Z ##[group]Sccache Compilation Log 2025-11-03T16:42:00.8298381Z + echo '=================== sccache compilation log ===================' 2025-11-03T16:42:00.8298657Z =================== sccache compilation log =================== 2025-11-03T16:42:00.8299053Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-11-03T16:42:00.8489213Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-11-03T16:42:00.8493337Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-11-03T16:42:00.8497508Z + sccache --show-stats 2025-11-03T16:42:00.8533447Z Compile requests 276 2025-11-03T16:42:00.8535324Z Compile requests executed 0 2025-11-03T16:42:00.8535564Z Cache hits 0 2025-11-03T16:42:00.8535772Z Cache misses 0 2025-11-03T16:42:00.8535964Z Cache hits rate - 2025-11-03T16:42:00.8536144Z Cache timeouts 0 2025-11-03T16:42:00.8536330Z Cache read errors 0 2025-11-03T16:42:00.8536512Z Forced recaches 0 2025-11-03T16:42:00.8536701Z Cache write errors 0 2025-11-03T16:42:00.8536874Z Cache errors 0 2025-11-03T16:42:00.8537065Z Compilations 0 2025-11-03T16:42:00.8537256Z Compilation failures 0 2025-11-03T16:42:00.8537450Z Non-cacheable compilations 0 2025-11-03T16:42:00.8537860Z Non-cacheable calls 25 2025-11-03T16:42:00.8538052Z Non-compilation calls 251 2025-11-03T16:42:00.8538247Z Unsupported compiler calls 0 2025-11-03T16:42:00.8538448Z Average cache write 0.000 s 2025-11-03T16:42:00.8538642Z Average compiler 0.000 s 2025-11-03T16:42:00.8538846Z Average cache read hit 0.000 s 2025-11-03T16:42:00.8539049Z Failed distributed compilations 0 2025-11-03T16:42:00.8539174Z 2025-11-03T16:42:00.8539251Z Non-cacheable reasons: 2025-11-03T16:42:00.8539422Z -E 25 2025-11-03T16:42:00.8539544Z 2025-11-03T16:42:00.8539691Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-11-03T16:42:00.8539962Z Version (client) 0.10.0 2025-11-03T16:42:00.8540294Z + sccache --stop-server 2025-11-03T16:42:00.8550310Z Stopping sccache server... 2025-11-03T16:42:00.8553886Z Compile requests 276 2025-11-03T16:42:00.8554266Z Compile requests executed 0 2025-11-03T16:42:00.8554498Z Cache hits 0 2025-11-03T16:42:00.8554713Z Cache misses 0 2025-11-03T16:42:00.8554951Z Cache hits rate - 2025-11-03T16:42:00.8555170Z Cache timeouts 0 2025-11-03T16:42:00.8555381Z Cache read errors 0 2025-11-03T16:42:00.8555585Z Forced recaches 0 2025-11-03T16:42:00.8555783Z Cache write errors 0 2025-11-03T16:42:00.8555979Z Cache errors 0 2025-11-03T16:42:00.8556186Z Compilations 0 2025-11-03T16:42:00.8556415Z Compilation failures 0 2025-11-03T16:42:00.8556815Z Non-cacheable compilations 0 2025-11-03T16:42:00.8557051Z Non-cacheable calls 25 2025-11-03T16:42:00.8557255Z Non-compilation calls 251 2025-11-03T16:42:00.8557456Z Unsupported compiler calls 0 2025-11-03T16:42:00.8557674Z Average cache write 0.000 s 2025-11-03T16:42:00.8557891Z Average compiler 0.000 s 2025-11-03T16:42:00.8558103Z Average cache read hit 0.000 s 2025-11-03T16:42:00.8558309Z Failed distributed compilations 0 2025-11-03T16:42:00.8558451Z 2025-11-03T16:42:00.8558576Z Non-cacheable reasons: 2025-11-03T16:42:00.8558759Z -E 25 2025-11-03T16:42:00.8558887Z 2025-11-03T16:42:00.8559056Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-11-03T16:42:00.8559355Z Version (client) 0.10.0 2025-11-03T16:42:00.8559596Z + echo ::endgroup:: 2025-11-03T16:42:00.8560139Z ##[endgroup] 2025-11-03T16:42:00.8560304Z + cleanup_workspace 2025-11-03T16:42:00.8560628Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-11-03T16:42:00.8561117Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-11-03T16:42:00.8561527Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-11-03T16:42:00.8561839Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-11-03T16:42:00.8562197Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-11-03T16:42:00.8562564Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-11-03T16:42:00.8562880Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-11-03T16:42:01.2717207Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-11-03T16:42:01.2717566Z with: 2025-11-03T16:42:01.2717795Z benchmark-results-dir: test/test-reports 2025-11-03T16:42:01.2718054Z dry-run: false 2025-11-03T16:42:01.2718263Z schema-version: v3 2025-11-03T16:42:01.2718715Z github-token: *** 2025-11-03T16:42:01.2718912Z env: 2025-11-03T16:42:01.2719119Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:01.2719599Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:01.2720005Z ##[endgroup] 2025-11-03T16:42:01.2748734Z ##[group]Run set -eux 2025-11-03T16:42:01.2748935Z set -eux 2025-11-03T16:42:01.2749075Z  2025-11-03T16:42:01.2749220Z if [[ -n "" ]]; then 2025-11-03T16:42:01.2749396Z  source "" 2025-11-03T16:42:01.2749558Z fi 2025-11-03T16:42:01.2749771Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-11-03T16:42:01.2750049Z  2025-11-03T16:42:01.2750188Z DEVICE_NAME="" 2025-11-03T16:42:01.2750346Z DEVICE_TYPE="" 2025-11-03T16:42:01.2750497Z  2025-11-03T16:42:01.2750648Z if command -v nvidia-smi; then 2025-11-03T16:42:01.2750914Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-11-03T16:42:01.2751235Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-11-03T16:42:01.2751550Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-11-03T16:42:01.2751804Z  python3 -mpip install torch==2.7.1 2025-11-03T16:42:01.2752015Z elif command -v rocminfo; then 2025-11-03T16:42:01.2752264Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-11-03T16:42:01.2752580Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-11-03T16:42:01.2752895Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-11-03T16:42:01.2753176Z  # GPU device name coming from rocminfo instead 2025-11-03T16:42:01.2753392Z  DEVICE_NAME=rocm 2025-11-03T16:42:01.2753670Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-11-03T16:42:01.2753948Z fi 2025-11-03T16:42:01.2754087Z  2025-11-03T16:42:01.2754391Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-11-03T16:42:01.2754650Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-11-03T16:42:01.2762366Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:01.2762594Z env: 2025-11-03T16:42:01.2762747Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:01.2763036Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:01.2763405Z ##[endgroup] 2025-11-03T16:42:01.2791565Z + [[ -n '' ]] 2025-11-03T16:42:01.2793569Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-11-03T16:42:01.4460818Z Defaulting to user installation because normal site-packages is not writeable 2025-11-03T16:42:02.1879147Z Collecting boto3==1.35.33 2025-11-03T16:42:02.2022899Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-11-03T16:42:02.4270712Z Collecting psutil==7.0.0 2025-11-03T16:42:02.4305875Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-11-03T16:42:02.4557627Z Collecting pynvml==12.0.0 2025-11-03T16:42:02.4593317Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-11-03T16:42:02.4949514Z Collecting s3transfer<0.11.0,>=0.10.0 2025-11-03T16:42:02.4981419Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-11-03T16:42:02.5037419Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-11-03T16:42:03.2989017Z Collecting botocore<1.36.0,>=1.35.33 2025-11-03T16:42:03.3022643Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-11-03T16:42:03.4517332Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-11-03T16:42:03.4552738Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-11-03T16:42:03.4642887Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-11-03T16:42:03.4644942Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-11-03T16:42:03.5730483Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-11-03T16:42:03.6717137Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-11-03T16:42:04.0019886Z Attempting uninstall: nvidia-ml-py 2025-11-03T16:42:04.0021515Z Found existing installation: nvidia-ml-py 11.525.84 2025-11-03T16:42:04.0029474Z Uninstalling nvidia-ml-py-11.525.84: 2025-11-03T16:42:04.0150665Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-11-03T16:42:04.0626471Z Attempting uninstall: psutil 2025-11-03T16:42:04.0627874Z Found existing installation: psutil 5.9.8 2025-11-03T16:42:04.0669872Z Uninstalling psutil-5.9.8: 2025-11-03T16:42:04.0671708Z Successfully uninstalled psutil-5.9.8 2025-11-03T16:42:04.1939268Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-11-03T16:42:04.3030935Z + DEVICE_NAME= 2025-11-03T16:42:04.3032416Z + DEVICE_TYPE= 2025-11-03T16:42:04.3032609Z + command -v nvidia-smi 2025-11-03T16:42:04.3032789Z + command -v rocminfo 2025-11-03T16:42:04.3032945Z + echo DEVICE_NAME= 2025-11-03T16:42:04.3033283Z + echo DEVICE_TYPE= 2025-11-03T16:42:04.3064358Z ##[group]Run set -eux 2025-11-03T16:42:04.3064546Z set -eux 2025-11-03T16:42:04.3064765Z  2025-11-03T16:42:04.3064939Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-11-03T16:42:04.3065158Z  echo "Missing github-token input" 2025-11-03T16:42:04.3065350Z  exit 1 2025-11-03T16:42:04.3065496Z fi 2025-11-03T16:42:04.3071782Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:04.3072015Z env: 2025-11-03T16:42:04.3072167Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:04.3072463Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:04.3072777Z DEVICE_NAME: 2025-11-03T16:42:04.3072923Z DEVICE_TYPE: 2025-11-03T16:42:04.3073323Z GITHUB_TOKEN: *** 2025-11-03T16:42:04.3073488Z ##[endgroup] 2025-11-03T16:42:04.3097051Z + [[ -z *** ]] 2025-11-03T16:42:04.3135794Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-11-03T16:42:04.3136134Z with: 2025-11-03T16:42:04.3136415Z github-token: *** 2025-11-03T16:42:04.3136571Z env: 2025-11-03T16:42:04.3136730Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:04.3137010Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:04.3137306Z DEVICE_NAME: 2025-11-03T16:42:04.3137460Z DEVICE_TYPE: 2025-11-03T16:42:04.3137611Z ##[endgroup] 2025-11-03T16:42:04.3157277Z ##[group]Run set -eux 2025-11-03T16:42:04.3157476Z set -eux 2025-11-03T16:42:04.3157637Z  2025-11-03T16:42:04.3157952Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-11-03T16:42:04.3162486Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:04.3162732Z env: 2025-11-03T16:42:04.3162879Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:04.3163158Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:04.3163447Z DEVICE_NAME: 2025-11-03T16:42:04.3163597Z DEVICE_TYPE: 2025-11-03T16:42:04.3163874Z GITHUB_TOKEN: *** 2025-11-03T16:42:04.3164168Z ##[endgroup] 2025-11-03T16:42:04.3186710Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 19040285537 i-0d3c8af4c7ead8235 2025-11-03T16:42:05.3689018Z setting job-id=54378387707 2025-11-03T16:42:05.3691387Z setting job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:42:05.3801793Z ##[group]Run set -eux 2025-11-03T16:42:05.3802075Z set -eux 2025-11-03T16:42:05.3802229Z  2025-11-03T16:42:05.3802380Z if [[ -n "" ]]; then 2025-11-03T16:42:05.3802567Z  source "" 2025-11-03T16:42:05.3802715Z fi 2025-11-03T16:42:05.3802856Z  2025-11-03T16:42:05.3803094Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-11-03T16:42:05.3803403Z  --schema-version "${SCHEMA_VERSION}" \ 2025-11-03T16:42:05.3803607Z  --repo "${REPO}" \ 2025-11-03T16:42:05.3803812Z  --head-branch "${HEAD_BRANCH}" \ 2025-11-03T16:42:05.3804018Z  --head-sha "${HEAD_SHA}" \ 2025-11-03T16:42:05.3804227Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-11-03T16:42:05.3804434Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-11-03T16:42:05.3804633Z  --job-id "${JOB_ID}" \ 2025-11-03T16:42:05.3804820Z  --job-name "${JOB_NAME}" 2025-11-03T16:42:05.3809315Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:05.3809542Z env: 2025-11-03T16:42:05.3809683Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:05.3809967Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:05.3810259Z DEVICE_NAME: 2025-11-03T16:42:05.3810409Z DEVICE_TYPE: 2025-11-03T16:42:05.3810553Z SCHEMA_VERSION: v3 2025-11-03T16:42:05.3810724Z REPO: pytorch/pytorch 2025-11-03T16:42:05.3810896Z HEAD_BRANCH: refs/heads/main 2025-11-03T16:42:05.3811104Z HEAD_SHA: 3f6538febd727b782e6e13cfd026a309fb14351d 2025-11-03T16:42:05.3811310Z WORKFLOW_RUN_ID: 19040285537 2025-11-03T16:42:05.3811478Z RUN_ATTEMPT: 1 2025-11-03T16:42:05.3811626Z JOB_ID: 54378387707 2025-11-03T16:42:05.3811902Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:42:05.3812184Z ##[endgroup] 2025-11-03T16:42:05.3840304Z + [[ -n '' ]] 2025-11-03T16:42:05.3843360Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 3f6538febd727b782e6e13cfd026a309fb14351d --workflow-id 19040285537 --run-attempt 1 --job-id 54378387707 --job-name 'inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-11-03T16:42:05.4094457Z ##[group]Run set -eux 2025-11-03T16:42:05.4094667Z set -eux 2025-11-03T16:42:05.4094818Z  2025-11-03T16:42:05.4094987Z if [[ -n "" ]]; then 2025-11-03T16:42:05.4095153Z  source "" 2025-11-03T16:42:05.4095303Z fi 2025-11-03T16:42:05.4095445Z  2025-11-03T16:42:05.4095685Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-11-03T16:42:05.4100018Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:05.4100236Z env: 2025-11-03T16:42:05.4100382Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:05.4100671Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:05.4100960Z DEVICE_NAME: 2025-11-03T16:42:05.4101101Z DEVICE_TYPE: 2025-11-03T16:42:05.4101251Z ##[endgroup] 2025-11-03T16:42:05.4124222Z + [[ -n '' ]] 2025-11-03T16:42:05.4126724Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-11-03T16:42:05.4430478Z INFO:root:Fail to import torch to get the device name 2025-11-03T16:42:05.4527458Z ##[group]Run set -eux 2025-11-03T16:42:05.4527650Z set -eux 2025-11-03T16:42:05.4527798Z  2025-11-03T16:42:05.4527957Z # TODO (huydhn): Implement this part 2025-11-03T16:42:05.4528193Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-11-03T16:42:05.4532534Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:05.4532759Z env: 2025-11-03T16:42:05.4532972Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:05.4533256Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:05.4533551Z DEVICE_NAME: 2025-11-03T16:42:05.4533700Z DEVICE_TYPE: 2025-11-03T16:42:05.4533848Z ##[endgroup] 2025-11-03T16:42:05.4554140Z + echo 'dependencies={}' 2025-11-03T16:42:05.4581715Z ##[group]Run set -eux 2025-11-03T16:42:05.4581923Z set -eux 2025-11-03T16:42:05.4582075Z  2025-11-03T16:42:05.4582218Z if [[ -n "" ]]; then 2025-11-03T16:42:05.4582409Z  source "" 2025-11-03T16:42:05.4582565Z fi 2025-11-03T16:42:05.4582717Z  2025-11-03T16:42:05.4582884Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-11-03T16:42:05.4583154Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-11-03T16:42:05.4583444Z  # We don't want the job to fail if the directory doesn't exist 2025-11-03T16:42:05.4583678Z  exit 0 2025-11-03T16:42:05.4583823Z fi 2025-11-03T16:42:05.4583961Z  2025-11-03T16:42:05.4584122Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-11-03T16:42:05.4584410Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-11-03T16:42:05.4584740Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-11-03T16:42:05.4584994Z  --metadata "${BENCHMARK_METADATA}" \ 2025-11-03T16:42:05.4585213Z  --runners "${RUNNER_INFO}" \ 2025-11-03T16:42:05.4585432Z  --dependencies "${DEPENDENCIES}" \ 2025-11-03T16:42:05.4585637Z  --dry-run 2025-11-03T16:42:05.4585790Z else 2025-11-03T16:42:05.4586023Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-11-03T16:42:05.4586341Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-11-03T16:42:05.4586591Z  --metadata "${BENCHMARK_METADATA}" \ 2025-11-03T16:42:05.4586797Z  --runners "${RUNNER_INFO}" \ 2025-11-03T16:42:05.4587007Z  --dependencies "${DEPENDENCIES}" 2025-11-03T16:42:05.4587200Z fi 2025-11-03T16:42:05.4590792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:05.4591013Z env: 2025-11-03T16:42:05.4591151Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:05.4591513Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:05.4591799Z DEVICE_NAME: 2025-11-03T16:42:05.4591948Z DEVICE_TYPE: 2025-11-03T16:42:05.4592114Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-11-03T16:42:05.4592309Z DRY_RUN: false 2025-11-03T16:42:05.4593042Z BENCHMARK_METADATA: {"timestamp": 1762188125, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "3f6538febd727b782e6e13cfd026a309fb14351d", "workflow_id": 19040285537, "run_attempt": 1, "job_id": 54378387707} 2025-11-03T16:42:05.4593955Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-34-62.ec2.internal"}, "name": "", "type": ""}] 2025-11-03T16:42:05.4594463Z DEPENDENCIES: {} 2025-11-03T16:42:05.4594632Z ##[endgroup] 2025-11-03T16:42:05.4616373Z + [[ -n '' ]] 2025-11-03T16:42:05.4618478Z + [[ ! -d test/test-reports ]] 2025-11-03T16:42:05.4618854Z + [[ false == \t\r\u\e ]] 2025-11-03T16:42:05.4625599Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1762188125, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "3f6538febd727b782e6e13cfd026a309fb14351d", "workflow_id": 19040285537, "run_attempt": 1, "job_id": 54378387707}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-34-62.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-11-03T16:42:05.5681850Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/19040285537/54378387707/inference_huggingface.json 2025-11-03T16:42:05.5935556Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-11-03T16:42:05.8328730Z ##[group]Run cat test/**/*_toprint.log || true 2025-11-03T16:42:05.8328992Z cat test/**/*_toprint.log || true 2025-11-03T16:42:05.8333552Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:05.8333782Z env: 2025-11-03T16:42:05.8333931Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:05.8334219Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:05.8334505Z DEVICE_NAME: 2025-11-03T16:42:05.8334668Z DEVICE_TYPE: 2025-11-03T16:42:05.8334813Z ##[endgroup] 2025-11-03T16:42:05.8409973Z cat: 'test/**/*_toprint.log': No such file or directory 2025-11-03T16:42:05.8446897Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-11-03T16:42:05.8447128Z kill "$MONITOR_SCRIPT_PID" 2025-11-03T16:42:05.8450815Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:05.8451062Z env: 2025-11-03T16:42:05.8451212Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:05.8451503Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:05.8451797Z DEVICE_NAME: 2025-11-03T16:42:05.8451949Z DEVICE_TYPE: 2025-11-03T16:42:05.8452106Z MONITOR_SCRIPT_PID: 48457 2025-11-03T16:42:05.8452280Z ##[endgroup] 2025-11-03T16:42:05.8545041Z Prepare all required actions 2025-11-03T16:42:05.8545352Z Getting action download info 2025-11-03T16:42:06.0027038Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-11-03T16:42:06.2381192Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-11-03T16:42:06.6709624Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-11-03T16:42:06.6709915Z with: 2025-11-03T16:42:06.6710244Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:42:06.6710729Z s3-bucket: gha-artifacts 2025-11-03T16:42:06.6710944Z env: 2025-11-03T16:42:06.6711133Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:06.6711528Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:06.6711923Z DEVICE_NAME: 2025-11-03T16:42:06.6712112Z DEVICE_TYPE: 2025-11-03T16:42:06.6712290Z ##[endgroup] 2025-11-03T16:42:06.6738325Z ##[group]Run # Remove any previous test jsons if they exist 2025-11-03T16:42:06.6738622Z # Remove any previous test jsons if they exist 2025-11-03T16:42:06.6738849Z rm -f test-jsons-*.zip 2025-11-03T16:42:06.6739116Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-11-03T16:42:06.6743687Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:06.6743912Z env: 2025-11-03T16:42:06.6744052Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:06.6744335Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:06.6744631Z DEVICE_NAME: 2025-11-03T16:42:06.6744781Z DEVICE_TYPE: 2025-11-03T16:42:06.6745035Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:42:06.6745313Z ##[endgroup] 2025-11-03T16:42:06.6927155Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-11-03T16:42:06.6956722Z ##[group]Run # Remove any previous test reports if they exist 2025-11-03T16:42:06.6957024Z # Remove any previous test reports if they exist 2025-11-03T16:42:06.6957259Z rm -f test-reports-*.zip 2025-11-03T16:42:06.6957547Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-11-03T16:42:06.6961593Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:06.6961821Z env: 2025-11-03T16:42:06.6961967Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:06.6962258Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:06.6962554Z DEVICE_NAME: 2025-11-03T16:42:06.6962708Z DEVICE_TYPE: 2025-11-03T16:42:06.6962971Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:42:06.6963262Z ##[endgroup] 2025-11-03T16:42:06.7015598Z adding: test/test-reports/inference_huggingface.csv (deflated 63%) 2025-11-03T16:42:06.7020478Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-11-03T16:42:06.7022665Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 64%) 2025-11-03T16:42:06.7047266Z ##[group]Run # Remove any previous usage logs if they exist 2025-11-03T16:42:06.7047578Z # Remove any previous usage logs if they exist 2025-11-03T16:42:06.7047822Z rm -f logs-*.zip 2025-11-03T16:42:06.7048063Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-11-03T16:42:06.7048375Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-11-03T16:42:06.7052116Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:06.7052342Z env: 2025-11-03T16:42:06.7052488Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:06.7052774Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:06.7053057Z DEVICE_NAME: 2025-11-03T16:42:06.7053203Z DEVICE_TYPE: 2025-11-03T16:42:06.7053572Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:42:06.7053853Z ##[endgroup] 2025-11-03T16:42:06.7111007Z adding: usage_log.txt (deflated 96%) 2025-11-03T16:42:06.7125694Z 2025-11-03T16:42:06.7132508Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip) 2025-11-03T16:42:06.7157039Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-11-03T16:42:06.7157435Z # Remove any previous debugging artifacts if they exist 2025-11-03T16:42:06.7157768Z rm -f debug-*.zip 2025-11-03T16:42:06.7158180Z if [ -d 'test/debug' ]; then 2025-11-03T16:42:06.7174606Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-11-03T16:42:06.7174894Z fi 2025-11-03T16:42:06.7178882Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:06.7179124Z env: 2025-11-03T16:42:06.7179281Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:06.7179583Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:06.7179885Z DEVICE_NAME: 2025-11-03T16:42:06.7180045Z DEVICE_TYPE: 2025-11-03T16:42:06.7180312Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707 2025-11-03T16:42:06.7180611Z ##[endgroup] 2025-11-03T16:42:06.7253221Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-11-03T16:42:06.7253425Z with: 2025-11-03T16:42:06.7253581Z s3-bucket: gha-artifacts 2025-11-03T16:42:06.7253791Z s3-prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:06.7254008Z retention-days: 14 2025-11-03T16:42:06.7254178Z if-no-files-found: warn 2025-11-03T16:42:06.7254341Z path: test-jsons-*.zip 2025-11-03T16:42:06.7254504Z name: artifact 2025-11-03T16:42:06.7254652Z region: us-east-1 2025-11-03T16:42:06.7254789Z env: 2025-11-03T16:42:06.7254930Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:06.7255219Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:06.7255510Z DEVICE_NAME: 2025-11-03T16:42:06.7255658Z DEVICE_TYPE: 2025-11-03T16:42:06.7255794Z ##[endgroup] 2025-11-03T16:42:06.9840515Z NOTE: s3-prefix specified, ignoring name parameter 2025-11-03T16:42:06.9842634Z With the provided path, there will be 1 file uploaded 2025-11-03T16:42:06.9843036Z Uploading to s3 prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:06.9871380Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:07.1463536Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:07.1639177Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-11-03T16:42:07.1639382Z with: 2025-11-03T16:42:07.1639531Z s3-bucket: gha-artifacts 2025-11-03T16:42:07.1639738Z s3-prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:07.1639946Z retention-days: 14 2025-11-03T16:42:07.1640118Z if-no-files-found: error 2025-11-03T16:42:07.1640293Z path: test-reports-*.zip 2025-11-03T16:42:07.1640457Z name: artifact 2025-11-03T16:42:07.1640607Z region: us-east-1 2025-11-03T16:42:07.1640746Z env: 2025-11-03T16:42:07.1640883Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:07.1641169Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:07.1641457Z DEVICE_NAME: 2025-11-03T16:42:07.1641593Z DEVICE_TYPE: 2025-11-03T16:42:07.1641734Z ##[endgroup] 2025-11-03T16:42:07.4481342Z NOTE: s3-prefix specified, ignoring name parameter 2025-11-03T16:42:07.4481856Z With the provided path, there will be 1 file uploaded 2025-11-03T16:42:07.4482310Z Uploading to s3 prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:07.4512885Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:07.5554904Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:07.5721908Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-11-03T16:42:07.5722121Z with: 2025-11-03T16:42:07.5722279Z s3-bucket: gha-artifacts 2025-11-03T16:42:07.5722487Z s3-prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:07.5722705Z retention-days: 14 2025-11-03T16:42:07.5722888Z if-no-files-found: ignore 2025-11-03T16:42:07.5723052Z path: logs-*.zip 2025-11-03T16:42:07.5723205Z name: artifact 2025-11-03T16:42:07.5723352Z region: us-east-1 2025-11-03T16:42:07.5723497Z env: 2025-11-03T16:42:07.5723630Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:07.5723913Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:07.5724256Z DEVICE_NAME: 2025-11-03T16:42:07.5724402Z DEVICE_TYPE: 2025-11-03T16:42:07.5724538Z ##[endgroup] 2025-11-03T16:42:07.8168118Z NOTE: s3-prefix specified, ignoring name parameter 2025-11-03T16:42:07.8170159Z With the provided path, there will be 1 file uploaded 2025-11-03T16:42:07.8170511Z Uploading to s3 prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:07.8194743Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:07.9521150Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:07.9697585Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-11-03T16:42:07.9697789Z with: 2025-11-03T16:42:07.9697942Z s3-bucket: gha-artifacts 2025-11-03T16:42:07.9698155Z s3-prefix: pytorch/pytorch/19040285537/1/artifact 2025-11-03T16:42:07.9698366Z retention-days: 14 2025-11-03T16:42:07.9698534Z if-no-files-found: ignore 2025-11-03T16:42:07.9698706Z path: debug-*.zip 2025-11-03T16:42:07.9698855Z name: artifact 2025-11-03T16:42:07.9699000Z region: us-east-1 2025-11-03T16:42:07.9699138Z env: 2025-11-03T16:42:07.9699277Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:07.9699575Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:07.9699861Z DEVICE_NAME: 2025-11-03T16:42:07.9699998Z DEVICE_TYPE: 2025-11-03T16:42:07.9700140Z ##[endgroup] 2025-11-03T16:42:08.2049771Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-11-03T16:42:08.2225832Z ##[group]Run # shellcheck disable=SC2156 2025-11-03T16:42:08.2226081Z # shellcheck disable=SC2156 2025-11-03T16:42:08.2226428Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-11-03T16:42:08.2231618Z shell: /usr/bin/bash -e {0} 2025-11-03T16:42:08.2231802Z env: 2025-11-03T16:42:08.2231945Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:08.2232222Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:08.2232516Z DEVICE_NAME: 2025-11-03T16:42:08.2232667Z DEVICE_TYPE: 2025-11-03T16:42:08.2232809Z ##[endgroup] 2025-11-03T16:42:08.3963525Z Prepare all required actions 2025-11-03T16:42:08.3963906Z Getting action download info 2025-11-03T16:42:08.5444403Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-11-03T16:42:08.5444638Z with: 2025-11-03T16:42:08.5444788Z job_id: 54378387707 2025-11-03T16:42:08.5445064Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:42:08.5445375Z workflow_name: inductor 2025-11-03T16:42:08.5445537Z workflow_run_id: 19040285537 2025-11-03T16:42:08.5445706Z workflow_attempt: 1 2025-11-03T16:42:08.5445855Z env: 2025-11-03T16:42:08.5445988Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:08.5446270Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:08.5446567Z DEVICE_NAME: 2025-11-03T16:42:08.5446716Z DEVICE_TYPE: 2025-11-03T16:42:08.5446860Z ##[endgroup] 2025-11-03T16:42:08.5505198Z ##[group]Run echo "workflow_id: 19040285537" 2025-11-03T16:42:08.5505461Z echo "workflow_id: 19040285537" 2025-11-03T16:42:08.5505668Z echo "workflow_attempt: 1" 2025-11-03T16:42:08.5505858Z echo "workflow_Name: inductor" 2025-11-03T16:42:08.5506052Z echo "job_id: 54378387707" 2025-11-03T16:42:08.5506375Z echo "job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" 2025-11-03T16:42:08.5506703Z echo "artifact_prefix: " 2025-11-03T16:42:08.5506893Z python3 --version 2025-11-03T16:42:08.5511349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:08.5511573Z env: 2025-11-03T16:42:08.5511719Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:08.5512066Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:08.5512353Z DEVICE_NAME: 2025-11-03T16:42:08.5512502Z DEVICE_TYPE: 2025-11-03T16:42:08.5512648Z ##[endgroup] 2025-11-03T16:42:08.5538217Z workflow_id: 19040285537 2025-11-03T16:42:08.5542795Z workflow_attempt: 1 2025-11-03T16:42:08.5546574Z workflow_Name: inductor 2025-11-03T16:42:08.5550637Z job_id: 54378387707 2025-11-03T16:42:08.5551005Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-11-03T16:42:08.5551376Z artifact_prefix: 2025-11-03T16:42:08.5551549Z Python 3.9.23 2025-11-03T16:42:08.5587171Z ##[group]Run nick-fields/retry@v3.0.0 2025-11-03T16:42:08.5587372Z with: 2025-11-03T16:42:08.5587508Z shell: bash 2025-11-03T16:42:08.5587661Z timeout_minutes: 5 2025-11-03T16:42:08.5587821Z max_attempts: 5 2025-11-03T16:42:08.5587980Z retry_wait_seconds: 30 2025-11-03T16:42:08.5588306Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-11-03T16:42:08.5588660Z polling_interval_seconds: 1 2025-11-03T16:42:08.5588853Z warning_on_retry: true 2025-11-03T16:42:08.5589024Z continue_on_error: false 2025-11-03T16:42:08.5589189Z env: 2025-11-03T16:42:08.5589335Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:08.5589616Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:08.5589918Z DEVICE_NAME: 2025-11-03T16:42:08.5590068Z DEVICE_TYPE: 2025-11-03T16:42:08.5590214Z ##[endgroup] 2025-11-03T16:42:08.8089171Z Defaulting to user installation because normal site-packages is not writeable 2025-11-03T16:42:08.8638920Z Collecting python-dateutil==2.8.2 2025-11-03T16:42:08.8760076Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-11-03T16:42:09.5298042Z Collecting boto3==1.35.42 2025-11-03T16:42:09.5330383Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-11-03T16:42:09.8968936Z Collecting pandas==2.1.3 2025-11-03T16:42:09.9012143Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-11-03T16:42:09.9854562Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-11-03T16:42:09.9865490Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-11-03T16:42:09.9898141Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-11-03T16:42:09.9900074Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-11-03T16:42:09.9906645Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-11-03T16:42:10.6208832Z Collecting numpy<2,>=1.22.4 2025-11-03T16:42:10.6242503Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-11-03T16:42:10.7578012Z Collecting tzdata>=2022.1 2025-11-03T16:42:10.7607767Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-11-03T16:42:10.7700550Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-11-03T16:42:10.7721059Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-11-03T16:42:10.7725661Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-11-03T16:42:10.7762638Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-11-03T16:42:10.7853854Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-11-03T16:42:10.7924976Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-11-03T16:42:10.7927051Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.15.0) 2025-11-03T16:42:10.9328404Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3 2025-11-03T16:42:14.6177819Z Attempting uninstall: boto3 2025-11-03T16:42:14.6179360Z Found existing installation: boto3 1.35.33 2025-11-03T16:42:14.6241009Z Uninstalling boto3-1.35.33: 2025-11-03T16:42:14.6251441Z Successfully uninstalled boto3-1.35.33 2025-11-03T16:42:14.6653812Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-11-03T16:42:15.6252244Z Command completed after 1 attempt(s). 2025-11-03T16:42:15.6314441Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-11-03T16:42:15.6314880Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-11-03T16:42:15.6315219Z  --workflow-run-id "19040285537" \ 2025-11-03T16:42:15.6315437Z  --workflow-name "inductor" \ 2025-11-03T16:42:15.6315649Z  --workflow-run-attempt "1" \ 2025-11-03T16:42:15.6315847Z  --job-id "54378387707" \ 2025-11-03T16:42:15.6316167Z  --job-name "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" \ 2025-11-03T16:42:15.6316556Z  --local-path "" \ 2025-11-03T16:42:15.6316746Z  --artifact-prefix "" 2025-11-03T16:42:15.6321367Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:15.6321597Z env: 2025-11-03T16:42:15.6321749Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:15.6322038Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:15.6322345Z DEVICE_NAME: 2025-11-03T16:42:15.6322500Z DEVICE_TYPE: 2025-11-03T16:42:15.6322646Z ##[endgroup] 2025-11-03T16:42:16.3960557Z repo: pytorch/pytorch 2025-11-03T16:42:16.3965426Z Search for test log in s3 bucket: ossci-utilization 2025-11-03T16:42:16.3967003Z Downloading logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:16.3967536Z extracting usage_log.txt from zip file logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_54378387707.zip 2025-11-03T16:42:16.3967927Z Converted Log Model: UtilizationMetadata: 2025-11-03T16:42:16.3968722Z UtilizationMetadata(level='metadata', workflow_id='19040285537', job_id='54378387707', workflow_name='inductor', job_name='inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1762187232, gpu_count=0, cpu_count=32, gpu_type=None, error=None) 2025-11-03T16:42:16.3969541Z [Db Segments] detected pytest cmd: 9, generated segments: 9 2025-11-03T16:42:16.3969784Z [db model] Peek db timeseries 2025-11-03T16:42:16.3969957Z :{ 2025-11-03T16:42:16.3970101Z "created_at": 1762188136, 2025-11-03T16:42:16.3970279Z "type": "utilization", 2025-11-03T16:42:16.3970445Z "tags": [ 2025-11-03T16:42:16.3970588Z "record" 2025-11-03T16:42:16.3970735Z ], 2025-11-03T16:42:16.3970876Z "time_stamp": 1762187232, 2025-11-03T16:42:16.3971058Z "repo": "pytorch/pytorch", 2025-11-03T16:42:16.3971240Z "workflow_id": 19040285537, 2025-11-03T16:42:16.3971443Z "run_attempt": 1, 2025-11-03T16:42:16.3971604Z "job_id": 54378387707, 2025-11-03T16:42:16.3971780Z "workflow_name": "inductor", 2025-11-03T16:42:16.3972085Z "job_name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", 2025-11-03T16:42:16.3972385Z "json_data": "{}" 2025-11-03T16:42:16.3972537Z } 2025-11-03T16:42:16.3972938Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/19040285537/1/54378387707/metadata 2025-11-03T16:42:16.3973448Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/19040285537/1/54378387707/metadata 2025-11-03T16:42:16.3973979Z Writing 178 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/19040285537/1/54378387707/time_series 2025-11-03T16:42:16.3974510Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/19040285537/1/54378387707/time_series 2025-11-03T16:42:16.4906518Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-11-03T16:42:16.4906799Z with: 2025-11-03T16:42:16.4906939Z env: 2025-11-03T16:42:16.4907077Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:16.4907369Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:16.4907664Z DEVICE_NAME: 2025-11-03T16:42:16.4907816Z DEVICE_TYPE: 2025-11-03T16:42:16.4907961Z ##[endgroup] 2025-11-03T16:42:16.4926035Z ##[group]Run set -eou pipefail 2025-11-03T16:42:16.4926428Z set -eou pipefail 2025-11-03T16:42:16.4926603Z  2025-11-03T16:42:16.4926830Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-11-03T16:42:16.4927100Z for _ in $(seq 1440); do 2025-11-03T16:42:16.4927313Z  # Break if no ssh session exists anymore 2025-11-03T16:42:16.4927524Z  if [ "$(who)" = "" ]; then 2025-11-03T16:42:16.4927707Z  break 2025-11-03T16:42:16.4927886Z  fi 2025-11-03T16:42:16.4928115Z  echo "." 2025-11-03T16:42:16.4928264Z  sleep 5 2025-11-03T16:42:16.4928415Z done 2025-11-03T16:42:16.4932774Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:16.4933000Z env: 2025-11-03T16:42:16.4933139Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:16.4933421Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:16.4933721Z DEVICE_NAME: 2025-11-03T16:42:16.4933868Z DEVICE_TYPE: 2025-11-03T16:42:16.4934008Z ##[endgroup] 2025-11-03T16:42:16.4958999Z Holding runner for 2 hours until all ssh sessions have logged out 2025-11-03T16:42:16.5044827Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-11-03T16:42:16.5045163Z # ignore expansion of "docker ps -q" since it could be empty 2025-11-03T16:42:16.5045423Z # shellcheck disable=SC2046 2025-11-03T16:42:16.5045646Z docker stop $(docker ps -q) || true 2025-11-03T16:42:16.5045855Z # Prune all of the docker images 2025-11-03T16:42:16.5046053Z docker system prune -af 2025-11-03T16:42:16.5049670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:16.5049897Z env: 2025-11-03T16:42:16.5050042Z GIT_DEFAULT_BRANCH: main 2025-11-03T16:42:16.5050319Z DOCKER_CONTAINER_ID: 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:16.5050616Z DEVICE_NAME: 2025-11-03T16:42:16.5050766Z DEVICE_TYPE: 2025-11-03T16:42:16.5050912Z ##[endgroup] 2025-11-03T16:42:27.3723153Z 06ea20fe7f0f 2025-11-03T16:42:27.6773012Z Deleted Containers: 2025-11-03T16:42:27.6773626Z 06ea20fe7f0f1e2e5ba61fd794af72b508d9113a93f7f88e9b037cf53081061a 2025-11-03T16:42:27.6774300Z 2025-11-03T16:42:34.5755202Z Deleted Images: 2025-11-03T16:42:34.5759915Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-30147fb242f88ab8f13e14b76ab186d189d8605c 2025-11-03T16:42:34.5761859Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:882a19f24dba11c5445e0a3a1cb4ef199b1740a1ba310ace5991cb025e30be3b 2025-11-03T16:42:34.5762526Z deleted: sha256:a0defb4d3830e67aa1d3adb1251bb453436eb10d09880acf2cba4d790dae265b 2025-11-03T16:42:34.5767205Z deleted: sha256:82ac7aec4f0bca562eab8c4825342b05cdafed8368784defc9e27f8791d0db2b 2025-11-03T16:42:34.5772529Z deleted: sha256:61ae75ee547c9fed8ba4e5630b635a83a5ed0193e2d6b0876370385c43cbc386 2025-11-03T16:42:34.5774553Z deleted: sha256:53e23ef5a6b5620bb3d374fd8f74b7771e84a127755907aeeba18bf7b17c7c08 2025-11-03T16:42:34.5775076Z deleted: sha256:b806f8084416a98d74c30a36c3fd5d4450605a892961a76b057b05fc9d6ebd33 2025-11-03T16:42:34.5779704Z deleted: sha256:cf7eb9f22f8db3b42689e81a452cdddc4a9d01a23859b5f5cd5194957dfd5bd7 2025-11-03T16:42:34.5781850Z deleted: sha256:28e1b7ba06f74fa7c83a873938efbb3584b900099fa4bd24ec15f4a7bb860cee 2025-11-03T16:42:34.5790632Z deleted: sha256:4031f9af404b9f2e23af1cd56e88fea15e6f849f4087bc18917c4c3e2a293817 2025-11-03T16:42:34.5791095Z deleted: sha256:76e95aa1d9201839a0f50b3fb107a816ae51d18940add18530952bee6ab5ea44 2025-11-03T16:42:34.5791472Z deleted: sha256:84ac6baf1e810689b526126263b31d507d06dd56b717485a2b896d5bd3a6801c 2025-11-03T16:42:34.5791851Z deleted: sha256:b8d5c6c7d66180b612f717fd1072abd5aac703aaa102291a5bb6058039531238 2025-11-03T16:42:34.5792236Z deleted: sha256:cd7a0db4b49b4911733be8b0d652ac8ec11a9d328e699b9a603cad1cecb2c152 2025-11-03T16:42:34.5792616Z deleted: sha256:208a055ef764eeb0753949b5753d2e515615de9853478916c64d841d9c818ad5 2025-11-03T16:42:34.5792984Z deleted: sha256:62150b30c29f90c98083ff2d70ae84a07b4a34df9d16a9d2dd414d0ddc164e57 2025-11-03T16:42:34.5793351Z deleted: sha256:0f7abc6f2c9060709f66e12ebdd4e439283b2365f46f0170a0728c67de3e2e21 2025-11-03T16:42:34.5793727Z deleted: sha256:ebe7cb1a3a2922e3cb9813507fd6dc93348eb0fe7c845732029ab2293b303057 2025-11-03T16:42:34.5794228Z deleted: sha256:f92e9acb12381f6060f8aace42598548176dd8bfb974b0e9d07e866889cc380b 2025-11-03T16:42:34.5794630Z deleted: sha256:01774ae2ba349772d5e9d98c5d03e8aee45f7a40817f904e8f0888520596974d 2025-11-03T16:42:34.5795107Z deleted: sha256:dd4aacf179ef25a5f6a71c6640b17f6cdfa9536d804c3fe05e8f668caf00b87e 2025-11-03T16:42:34.5795486Z deleted: sha256:35330dba33895bfdd998d21c2eb95b32097bc863d08ec6f9b2713689e1412763 2025-11-03T16:42:34.5795861Z deleted: sha256:b6e8897d94d19d8e5061b163b8b3fab4072d6df3a021775e9dc7bac6242a5236 2025-11-03T16:42:34.5796266Z deleted: sha256:b62627949005abd5a68f3e92bac6a6c555cf2c585c80008762c14d790f917a57 2025-11-03T16:42:34.5796646Z deleted: sha256:7b690d5476812bea1cdf504b61d2d1cf6378d65115de2ebc48a9ea3cab432cac 2025-11-03T16:42:34.5797071Z deleted: sha256:17afde0042e4353e06524353faae3c5314334a2071d52acbf038746118884443 2025-11-03T16:42:34.5797442Z deleted: sha256:67c49432bef88ff63042210d551b734824255efc2ec0034c805f3c923d38c9d7 2025-11-03T16:42:34.5797821Z deleted: sha256:6fbe2628d6c9f4307c7d193652e9d9a9897b8594c2f50a151e86e956b16bfb0e 2025-11-03T16:42:34.5798195Z deleted: sha256:ae4687b17b9d13098147b63064373278529539f1cb1455f8c2e141c2cb67678a 2025-11-03T16:42:34.5798575Z deleted: sha256:ecb6dbc02f6f686a59ba8cb5ffac62a0efea585941976221c6b6580c844f1dad 2025-11-03T16:42:34.5798962Z deleted: sha256:e7b8ee901c7e1ad0cab106a2fc7f2886a8efc5517d39822db601c1d9b7bcca2a 2025-11-03T16:42:34.5799337Z deleted: sha256:25c2f9556b1a7cc3bb18d2430898188693468fb55f2956c8198ef547baf3dcbf 2025-11-03T16:42:34.5799715Z deleted: sha256:f3a9fb1e4b0c62e9a5038472b38af0aae70bc68ae9135a810b243921e3988d25 2025-11-03T16:42:34.5800093Z deleted: sha256:9835b6a41a9c069ef208c7f44dd89c3fc60227bda774ced2f201ad1c0c095764 2025-11-03T16:42:34.5800467Z deleted: sha256:59ccb6478f5041bbf13a6a9c55efac0e8129dbc76b33a30a999f8c96494e5b06 2025-11-03T16:42:34.5800846Z deleted: sha256:9dbdd6727f46d4c586df4c452f22d6236bb5f525d78be86801b0e1ba3f660437 2025-11-03T16:42:34.5801221Z deleted: sha256:64fc2cc156c361a33e0433a6e05274ffb847d39e2c383acc0b64b6d86836113b 2025-11-03T16:42:34.5801599Z deleted: sha256:0d1e5b8a20a8351464cd5c70a407b6222f4dccee7fffc2800b55fce9e87a3272 2025-11-03T16:42:34.5801987Z deleted: sha256:c15d099db094c4ce7377d37bfdb7030b8a29ac7e01a224faaf68c6a34a64088f 2025-11-03T16:42:34.5802362Z deleted: sha256:577af5bfa5dad1cd57c52e7d25928ac02e46e5a7f7e7b3755879b88411efa364 2025-11-03T16:42:34.5802732Z deleted: sha256:2c552e56af39ee1ce57a188082fc3741975e8e930b492dd540f444c9af71822b 2025-11-03T16:42:34.5803107Z deleted: sha256:484814563b2babcb6d7d290972d0945c09ed773cf807ae77e93d309bac172b2d 2025-11-03T16:42:34.5803494Z deleted: sha256:6385deb53b9f6309f2683a0faf0702806f87d358f229dbc02441809d8aaec291 2025-11-03T16:42:34.5803850Z deleted: sha256:a2308737c8326ee8536a4b7b62d904b6ed8a1a8864d6a1be84b4bd9c80ece853 2025-11-03T16:42:34.5804212Z deleted: sha256:65b65732df68a535dbae91748965a60a841c6caced2df12407d3024036ca6823 2025-11-03T16:42:34.5804574Z deleted: sha256:14710f315b934b0068b6c395e5c64d6da8fe5dd4d691e1be713f5e3581774e9f 2025-11-03T16:42:34.5805002Z deleted: sha256:c32b0c1e6d822346f60ba3a34ae32b36e900b29848d4adf18ef1c845361abc6b 2025-11-03T16:42:34.5805365Z deleted: sha256:85151972074d1304dee07db913f717883702f736c391980185efdd274bcb15e5 2025-11-03T16:42:34.5805714Z deleted: sha256:b08d790bfff71a169f3b24099350138102506e8ac03c5302cd2913a67c6927e9 2025-11-03T16:42:34.5806084Z deleted: sha256:70b04de2e2ecca6134e6fc488ae17ddfe15fa14363584f38ab77ed41c2c5eb8c 2025-11-03T16:42:34.5806457Z deleted: sha256:f107a2cf62b8234a57c9dbabef97c64802c0b5b75a0dc34addbd9180d807781b 2025-11-03T16:42:34.5806825Z deleted: sha256:26921a3710106c63857fae410c65936c9d57a41b3d3ee792b712ce14cb400094 2025-11-03T16:42:34.5807184Z deleted: sha256:c52bf37890136cc6bd0ec54c3b48cfcd86bdca8b89c9f37a66f00dd8cdb85dbc 2025-11-03T16:42:34.5807551Z deleted: sha256:4c084c773d87ac6a7b53fd3b68dc57e72858598adda064b9be427151ce4296fc 2025-11-03T16:42:34.5807915Z deleted: sha256:0314cbf4ae7da0d9b65b7c8dac180e92a292321bd0580d45040bd50c0d0bdab1 2025-11-03T16:42:34.5808380Z deleted: sha256:9d90c078393230c96254979c79d7b59ac61722883e62404e6989f911c650bece 2025-11-03T16:42:34.5808747Z deleted: sha256:a607c4e66af475e5da6c7ed4554ffb1b91c626a8d0be96cfbd60d69cde21bfc4 2025-11-03T16:42:34.5809137Z deleted: sha256:22805768ac6564085909c7523b28cf5f430e6e25246381e9c8246d032085f447 2025-11-03T16:42:34.5809485Z deleted: sha256:fd0d7864865a14b3a03d8a07968315537bdf440fa069c908aeaedd6ec9f6b08d 2025-11-03T16:42:34.5809840Z deleted: sha256:40951534f06a293a8437e3ff21aa4092224f42308a3f6f4c4fe0121dc96e5db5 2025-11-03T16:42:34.5810203Z deleted: sha256:d7af1ad1a6660e8e1a2a16e691aab3e685bdbc43776dbcdc2078c11ca172bf82 2025-11-03T16:42:34.5810572Z deleted: sha256:37f5c0c751b9d6af825bf9a59ff92d14010a2203199eae7d09d2719fdaeb05f7 2025-11-03T16:42:34.5810927Z deleted: sha256:692b929e76151fdb96a95e51e889695c9e8b4d8bc64b185a004cbb086af8e018 2025-11-03T16:42:34.5811286Z deleted: sha256:4eaf11d355fc12616b65790f9844334fa31fa9c8e300c0ff66e7ba5b22bd6e50 2025-11-03T16:42:34.5811650Z deleted: sha256:15f6d4def1ce5d48a3bd64c8a1692dfdb53e9853830e01125d42d3ccfed48fc8 2025-11-03T16:42:34.5812017Z deleted: sha256:0f594ce1f8c39e40967bb7c66cc1dc9e5b724d612902f0dd3d5648816e2bdf75 2025-11-03T16:42:34.5812386Z deleted: sha256:b29d5650f65ff3c479680efda6adc93bcc9c533a340a86ad723cdc76e9a298a3 2025-11-03T16:42:34.5812749Z deleted: sha256:f292d461d3c679728d589e33eedb7ae89c1bdd16fd6237aaeb289bb1feacb1b6 2025-11-03T16:42:34.5813117Z deleted: sha256:669f78efd8a92cf04a59941047bee63928e770a10be40f289c2c01554a877ad6 2025-11-03T16:42:34.5813664Z deleted: sha256:0610e32004d9778a18d7c3198cebe1a21261ca0a22bc896cc02075390e371c4c 2025-11-03T16:42:34.5814021Z deleted: sha256:b34a3a5187a31b74c309baa927b3648f9498191210f15562eb8bd4b9c459d4bc 2025-11-03T16:42:34.5814379Z deleted: sha256:9f67593f5b06e110bc2b0889317497c272b98172aa91b8d51ed9d984620d06b3 2025-11-03T16:42:34.5814787Z deleted: sha256:3a1dd5381684672e726fa22fe5f52b4ff2a3221be8387bae31cac48d8cf44160 2025-11-03T16:42:34.5815138Z deleted: sha256:0762a25793757de2556a7b814d486a5a873ec574ac61f9d91a474eaf375e1b67 2025-11-03T16:42:34.5815494Z deleted: sha256:5bd16286d6de50db41e88d08de4dd16e9f29b7fde27240e7150b555eb6304032 2025-11-03T16:42:34.5815855Z deleted: sha256:f51568a24cfe465ba35d0d865d7e32f24c5a347c27192d67dc9520a9024ddf03 2025-11-03T16:42:34.5816196Z deleted: sha256:c0e8a0889134ef4a1266f4366a249c57d2d793559c6a6d97574717d5495c7d11 2025-11-03T16:42:34.5816545Z deleted: sha256:904a0c05776cc17071df74937da4e208fc2416280593996f81d3e841e1c41048 2025-11-03T16:42:34.5816894Z deleted: sha256:f06d72f7ba722688f62652416669b4724c8e30b74891813b455efdffe1f93466 2025-11-03T16:42:34.5817278Z deleted: sha256:0d4e7a386bc94313f389eadd45f688a707d1b5d4994378381450de2306a82c79 2025-11-03T16:42:34.5817628Z deleted: sha256:63cc9363306d393f09857fa7d93605c66c884fdcbbb254c37474b6cf4f293a35 2025-11-03T16:42:34.5817969Z deleted: sha256:767e56ba346ae714b6e6b816baa839051145ed78cfa0e4524a86cc287b0c4b00 2025-11-03T16:42:34.5818180Z 2025-11-03T16:42:34.5818261Z Total reclaimed space: 53.73GB 2025-11-03T16:42:34.5871596Z Post job cleanup. 2025-11-03T16:42:34.5914773Z Post job cleanup. 2025-11-03T16:42:34.6653744Z [command]/usr/bin/git version 2025-11-03T16:42:34.6693726Z git version 2.50.1 2025-11-03T16:42:34.6729272Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/a0387573-deeb-4ec1-956e-951b8cdeffa6/.gitconfig' 2025-11-03T16:42:34.6756728Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a0387573-deeb-4ec1-956e-951b8cdeffa6' before making global git config changes 2025-11-03T16:42:34.6759719Z Adding repository directory to the temporary git global config as a safe directory 2025-11-03T16:42:34.6764590Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-11-03T16:42:34.6803711Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-11-03T16:42:34.6835123Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-11-03T16:42:34.7152520Z Entering 'android/libs/fbjni' 2025-11-03T16:42:34.7209084Z Entering 'third_party/FP16' 2025-11-03T16:42:34.7266448Z Entering 'third_party/FXdiv' 2025-11-03T16:42:34.7324321Z Entering 'third_party/NNPACK' 2025-11-03T16:42:34.7376309Z Entering 'third_party/NVTX' 2025-11-03T16:42:34.7430920Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:42:34.7485818Z Entering 'third_party/XNNPACK' 2025-11-03T16:42:34.7551311Z Entering 'third_party/aiter' 2025-11-03T16:42:34.7603189Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:42:34.7665655Z Entering 'third_party/benchmark' 2025-11-03T16:42:34.7717999Z Entering 'third_party/composable_kernel' 2025-11-03T16:42:34.7775280Z Entering 'third_party/cpp-httplib' 2025-11-03T16:42:34.7830254Z Entering 'third_party/cpuinfo' 2025-11-03T16:42:34.7883451Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:42:34.7940909Z Entering 'third_party/cutlass' 2025-11-03T16:42:34.8001459Z Entering 'third_party/fbgemm' 2025-11-03T16:42:34.8056991Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:42:34.8112736Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:42:34.8174446Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:42:34.8223787Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:42:34.8285077Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:42:34.8338916Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:42:34.8392021Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:42:34.8456243Z Entering 'third_party/flash-attention' 2025-11-03T16:42:34.8509526Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:42:34.8568620Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:42:34.8632911Z Entering 'third_party/flatbuffers' 2025-11-03T16:42:34.8689215Z Entering 'third_party/fmt' 2025-11-03T16:42:34.8745916Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:42:34.8801074Z Entering 'third_party/gloo' 2025-11-03T16:42:34.8853721Z Entering 'third_party/googletest' 2025-11-03T16:42:34.8907587Z Entering 'third_party/ideep' 2025-11-03T16:42:34.8963433Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:42:34.9026186Z Entering 'third_party/ittapi' 2025-11-03T16:42:34.9078102Z Entering 'third_party/kineto' 2025-11-03T16:42:34.9131134Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:42:34.9184165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:42:34.9234360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:42:34.9287534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:42:34.9339356Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:42:34.9389001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:42:34.9448385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:42:34.9503892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:42:34.9556766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:42:34.9609453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:42:34.9664798Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:42:34.9722352Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:42:34.9776000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:42:34.9835815Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:42:34.9891894Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:42:34.9949953Z Entering 'third_party/kleidiai' 2025-11-03T16:42:35.0007082Z Entering 'third_party/mimalloc' 2025-11-03T16:42:35.0059423Z Entering 'third_party/nlohmann' 2025-11-03T16:42:35.0112343Z Entering 'third_party/onnx' 2025-11-03T16:42:35.0180317Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:42:35.0242946Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:42:35.0294706Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:42:35.0349154Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:42:35.0396943Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:42:35.0459175Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:42:35.0510315Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:42:35.0559223Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:42:35.0615288Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:42:35.0664208Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:42:35.0717206Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:42:35.0772922Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:42:35.0845316Z Entering 'third_party/pocketfft' 2025-11-03T16:42:35.0898187Z Entering 'third_party/protobuf' 2025-11-03T16:42:35.0951830Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:42:35.1009214Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:42:35.1068994Z Entering 'third_party/psimd' 2025-11-03T16:42:35.1126175Z Entering 'third_party/pthreadpool' 2025-11-03T16:42:35.1179619Z Entering 'third_party/pybind11' 2025-11-03T16:42:35.1233233Z Entering 'third_party/python-peachpy' 2025-11-03T16:42:35.1287932Z Entering 'third_party/sleef' 2025-11-03T16:42:35.1347634Z Entering 'third_party/tensorpipe' 2025-11-03T16:42:35.1398907Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:42:35.1451080Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:42:35.1502678Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:42:35.1560964Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:42:35.1612430Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:42:35.1691386Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-11-03T16:42:35.1720486Z http.https://github.com/.extraheader 2025-11-03T16:42:35.1733737Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-11-03T16:42:35.1769176Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-11-03T16:42:35.2081211Z Entering 'android/libs/fbjni' 2025-11-03T16:42:35.2115527Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2150082Z Entering 'third_party/FP16' 2025-11-03T16:42:35.2182013Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2219562Z Entering 'third_party/FXdiv' 2025-11-03T16:42:35.2254855Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2289672Z Entering 'third_party/NNPACK' 2025-11-03T16:42:35.2323983Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2360911Z Entering 'third_party/NVTX' 2025-11-03T16:42:35.2393442Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2428738Z Entering 'third_party/VulkanMemoryAllocator' 2025-11-03T16:42:35.2461569Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2493905Z Entering 'third_party/XNNPACK' 2025-11-03T16:42:35.2528785Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2575407Z Entering 'third_party/aiter' 2025-11-03T16:42:35.2608357Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2646237Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-11-03T16:42:35.2680812Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2726983Z Entering 'third_party/benchmark' 2025-11-03T16:42:35.2759228Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2796427Z Entering 'third_party/composable_kernel' 2025-11-03T16:42:35.2829528Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2871964Z Entering 'third_party/cpp-httplib' 2025-11-03T16:42:35.2908637Z http.https://github.com/.extraheader 2025-11-03T16:42:35.2943218Z Entering 'third_party/cpuinfo' 2025-11-03T16:42:35.2974986Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3009323Z Entering 'third_party/cudnn_frontend' 2025-11-03T16:42:35.3044285Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3078889Z Entering 'third_party/cutlass' 2025-11-03T16:42:35.3110114Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3153822Z Entering 'third_party/fbgemm' 2025-11-03T16:42:35.3187367Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3230109Z Entering 'third_party/fbgemm/external/asmjit' 2025-11-03T16:42:35.3258201Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3295309Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-11-03T16:42:35.3327067Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3368823Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-11-03T16:42:35.3399777Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3440191Z Entering 'third_party/fbgemm/external/cutlass' 2025-11-03T16:42:35.3470639Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3511789Z Entering 'third_party/fbgemm/external/googletest' 2025-11-03T16:42:35.3547277Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3584441Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-11-03T16:42:35.3621236Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3658386Z Entering 'third_party/fbgemm/external/json' 2025-11-03T16:42:35.3692913Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3733955Z Entering 'third_party/flash-attention' 2025-11-03T16:42:35.3769037Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3805015Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-11-03T16:42:35.3838412Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3885227Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-11-03T16:42:35.3920653Z http.https://github.com/.extraheader 2025-11-03T16:42:35.3961608Z Entering 'third_party/flatbuffers' 2025-11-03T16:42:35.3995218Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4032497Z Entering 'third_party/fmt' 2025-11-03T16:42:35.4065544Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4107002Z Entering 'third_party/gemmlowp/gemmlowp' 2025-11-03T16:42:35.4140313Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4179413Z Entering 'third_party/gloo' 2025-11-03T16:42:35.4211427Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4245475Z Entering 'third_party/googletest' 2025-11-03T16:42:35.4280390Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4313670Z Entering 'third_party/ideep' 2025-11-03T16:42:35.4348561Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4380426Z Entering 'third_party/ideep/mkl-dnn' 2025-11-03T16:42:35.4412664Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4456079Z Entering 'third_party/ittapi' 2025-11-03T16:42:35.4489576Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4525811Z Entering 'third_party/kineto' 2025-11-03T16:42:35.4561454Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4595936Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-11-03T16:42:35.4631133Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4665283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-11-03T16:42:35.4698241Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4734293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-11-03T16:42:35.4766958Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4805630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-11-03T16:42:35.4835838Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4873961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-11-03T16:42:35.4909055Z http.https://github.com/.extraheader 2025-11-03T16:42:35.4943047Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-11-03T16:42:35.4975959Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5021062Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-11-03T16:42:35.5051008Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5088768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-11-03T16:42:35.5123409Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5158985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-11-03T16:42:35.5192126Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5228901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-11-03T16:42:35.5269508Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5305188Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-11-03T16:42:35.5341141Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5371639Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:42:35.5404397Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5445848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:42:35.5479210Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5525622Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-11-03T16:42:35.5555164Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5592868Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-11-03T16:42:35.5625777Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5664727Z Entering 'third_party/kleidiai' 2025-11-03T16:42:35.5699313Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5736610Z Entering 'third_party/mimalloc' 2025-11-03T16:42:35.5771453Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5807685Z Entering 'third_party/nlohmann' 2025-11-03T16:42:35.5841769Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5882188Z Entering 'third_party/onnx' 2025-11-03T16:42:35.5918358Z http.https://github.com/.extraheader 2025-11-03T16:42:35.5966571Z Entering 'third_party/onnx/third_party/pybind11' 2025-11-03T16:42:35.6002852Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6040047Z Entering 'third_party/opentelemetry-cpp' 2025-11-03T16:42:35.6071428Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6109523Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-11-03T16:42:35.6144995Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6180710Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-11-03T16:42:35.6212539Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6250231Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-11-03T16:42:35.6282634Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6321563Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-11-03T16:42:35.6352901Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6392547Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-11-03T16:42:35.6427519Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6464334Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-11-03T16:42:35.6494881Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6534320Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-11-03T16:42:35.6565986Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6606439Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-11-03T16:42:35.6641336Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6675672Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-11-03T16:42:35.6710616Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6750885Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-11-03T16:42:35.6783560Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6837050Z Entering 'third_party/pocketfft' 2025-11-03T16:42:35.6866453Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6907669Z Entering 'third_party/protobuf' 2025-11-03T16:42:35.6940563Z http.https://github.com/.extraheader 2025-11-03T16:42:35.6977502Z Entering 'third_party/protobuf/third_party/benchmark' 2025-11-03T16:42:35.7009777Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7047337Z Entering 'third_party/protobuf/third_party/googletest' 2025-11-03T16:42:35.7079341Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7117868Z Entering 'third_party/psimd' 2025-11-03T16:42:35.7158270Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7194315Z Entering 'third_party/pthreadpool' 2025-11-03T16:42:35.7228687Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7264894Z Entering 'third_party/pybind11' 2025-11-03T16:42:35.7299128Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7335111Z Entering 'third_party/python-peachpy' 2025-11-03T16:42:35.7370137Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7407715Z Entering 'third_party/sleef' 2025-11-03T16:42:35.7442825Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7481292Z Entering 'third_party/tensorpipe' 2025-11-03T16:42:35.7519251Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7552203Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-11-03T16:42:35.7584056Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7623194Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-11-03T16:42:35.7655143Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7693492Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-11-03T16:42:35.7727581Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7764257Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-11-03T16:42:35.7797673Z http.https://github.com/.extraheader 2025-11-03T16:42:35.7829568Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-11-03T16:42:35.7863645Z http.https://github.com/.extraheader 2025-11-03T16:42:35.8005800Z A job completed hook has been configured by the self-hosted runner administrator 2025-11-03T16:42:35.8032035Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-11-03T16:42:35.8035649Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-11-03T16:42:35.8035901Z ##[endgroup] 2025-11-03T16:42:35.8112639Z [!ALERT!] Swap in detected! [!ALERT!] 2025-11-03T16:42:44.0668948Z [!ALERT!] Swap out detected [!ALERT!] 2025-11-03T16:42:58.1756805Z Cleaning up orphan processes